[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 764865713 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003220] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008635] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011010] pid_max: default: 32768 minimum: 301 [ 0.012129] LSM: Security Framework initializing [ 0.013051] Yama: becoming mindful. [ 0.014026] SELinux: Initializing. [ 0.015064] *** VALIDATE selinux *** [ 0.025623] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029713] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031082] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033106] *** VALIDATE tmpfs *** [ 0.035370] *** VALIDATE proc *** [ 0.036248] *** VALIDATE cgroup *** [ 0.037009] *** VALIDATE cgroup2 *** [ 0.038260] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039228] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041033] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.045564] debug: unmapping init [mem 0xffffffffbc259000-0xffffffffbc260fff] [ 0.048137] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049818] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050063] ... version: 2 [ 0.051037] ... bit width: 48 [ 0.052012] ... generic registers: 4 [ 0.053012] ... value mask: 0000ffffffffffff [ 0.054015] ... max period: 00007fffffffffff [ 0.055192] ... fixed-purpose events: 3 [ 0.056014] ... event mask: 000000070000000f [ 0.057261] rcu: Hierarchical SRCU implementation. [ 0.059896] smp: Bringing up secondary CPUs ... [ 0.060830] x86: Booting SMP configuration: [ 0.061026] .... node #0, CPUs: #1 #2 #3 [ 0.070273] smp: Brought up 1 node, 4 CPUs [ 0.072025] smpboot: Max logical packages: 1 [ 0.073015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.115731] node 0 deferred pages initialised in 41ms [ 0.120507] devtmpfs: initialized [ 0.121380] x86/mm: Memory block size: 128MB [ 0.124228] gcov: version magic: 0x41383552 [ 0.126468] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.127311] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.128309] pinctrl core: initialized pinctrl subsystem [ 0.129204] [ 0.130011] ************************************************************* [ 0.131013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.132013] ** ** [ 0.133014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.134015] ** ** [ 0.135023] ** This means that this kernel is built to expose internal ** [ 0.136015] ** IOMMU data structures, which may compromise security on ** [ 0.137013] ** your system. ** [ 0.138014] ** ** [ 0.139021] ** If you see this message and you are not debugging the ** [ 0.140013] ** kernel, report this immediately to your vendor! ** [ 0.141015] ** ** [ 0.142012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143016] ************************************************************* [ 0.144846] NET: Registered protocol family 16 [ 0.145649] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.146134] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.147231] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.149140] cpuidle: using governor menu [ 0.153454] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.158522] PCI: Using configuration type 1 for base access [ 0.161182] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.174186] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.181048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188435] cryptd: max_cpu_qlen set to 1000 [ 0.192788] ACPI: Added _OSI(Module Device) [ 0.194025] ACPI: Added _OSI(Processor Device) [ 0.201015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206046] ACPI: Added _OSI(Processor Aggregator Device) [ 0.223414] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.232593] ACPI: Interpreter enabled [ 0.233066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.234029] ACPI: Using IOAPIC for interrupt routing [ 0.235149] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.236537] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.248817] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.252043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.255017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.259074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.265917] acpiphp: Slot [2] registered [ 0.267157] acpiphp: Slot [5] registered [ 0.268000] acpiphp: Slot [6] registered [ 0.268000] acpiphp: Slot [7] registered [ 0.268000] acpiphp: Slot [8] registered [ 0.272075] acpiphp: Slot [9] registered [ 0.273127] acpiphp: Slot [10] registered [ 0.276112] acpiphp: Slot [3] registered [ 0.278255] acpiphp: Slot [4] registered [ 0.281307] acpiphp: Slot [11] registered [ 0.285320] acpiphp: Slot [12] registered [ 0.289280] acpiphp: Slot [13] registered [ 0.292130] acpiphp: Slot [14] registered [ 0.293844] acpiphp: Slot [15] registered [ 0.297176] acpiphp: Slot [16] registered [ 0.299184] acpiphp: Slot [17] registered [ 0.302158] acpiphp: Slot [18] registered [ 0.305267] acpiphp: Slot [19] registered [ 0.309270] acpiphp: Slot [20] registered [ 0.312118] acpiphp: Slot [21] registered [ 0.314202] acpiphp: Slot [22] registered [ 0.318198] acpiphp: Slot [23] registered [ 0.322222] acpiphp: Slot [24] registered [ 0.326349] acpiphp: Slot [25] registered [ 0.330295] acpiphp: Slot [26] registered [ 0.334460] acpiphp: Slot [27] registered [ 0.340422] acpiphp: Slot [28] registered [ 0.343354] acpiphp: Slot [29] registered [ 0.347170] acpiphp: Slot [30] registered [ 0.348000] acpiphp: Slot [31] registered [ 0.348000] PCI host bridge to bus 0000:00 [ 0.351021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.354023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.357024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.361024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.364023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.367025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.369252] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.372140] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.375431] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.385855] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.390069] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.393020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.395090] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.398019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.402632] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.406057] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.409044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.412011] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.417013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.427011] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.432018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.435641] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.450015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.460014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.495015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.513542] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.530019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.553015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.580015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.606409] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.620016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.627015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.658014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.673837] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.686016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.699020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.723021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.738189] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.750015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.762015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.792015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.813340] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.822015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.828019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.866015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.882932] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.885443] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.888379] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.891449] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.893205] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.899317] iommu: Default domain type: Passthrough [ 0.901425] SCSI subsystem initialized [ 0.902157] ACPI: bus type USB registered [ 0.903259] usbcore: registered new interface driver usbfs [ 0.905092] usbcore: registered new interface driver hub [ 0.907095] usbcore: registered new device driver usb [ 0.908178] pps_core: LinuxPPS API ver. 1 registered [ 0.910011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.914111] PTP clock support registered [ 0.917106] EDAC MC: Ver: 3.0.0 [ 0.921035] PCI: Using ACPI for IRQ routing [ 0.924912] NetLabel: Initializing [ 0.926009] NetLabel: domain hash size = 128 [ 0.928008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.930087] NetLabel: unlabeled traffic allowed by default [ 0.932340] vgaarb: loaded [ 0.934344] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.936014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.944649] clocksource: Switched to clocksource kvm-clock [ 1.084309] VFS: Disk quotas dquot_6.6.0 [ 1.085848] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.088071] *** VALIDATE ramfs *** [ 1.089099] *** VALIDATE hugetlbfs *** [ 1.090748] pnp: PnP ACPI init [ 1.092745] pnp: PnP ACPI: found 6 devices [ 1.108630] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.111685] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.114296] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.116709] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.118906] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.121352] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.124046] NET: Registered protocol family 2 [ 1.126405] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.132360] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.135879] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.141225] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.144797] TCP: Hash tables configured (established 65536 bind 65536) [ 1.147772] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.151202] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.154846] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.158330] NET: Registered protocol family 1 [ 1.161127] RPC: Registered named UNIX socket transport module. [ 1.163250] RPC: Registered udp transport module. [ 1.164248] RPC: Registered tcp transport module. [ 1.165482] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.167363] NET: Registered protocol family 44 [ 1.168563] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.170290] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.171922] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.173715] PCI: CLS 0 bytes, default 64 [ 1.174871] Unpacking initramfs... [ 2.731788] debug: unmapping init [mem 0xffff99bbbcc54000-0xffff99bbbffbffff] [ 2.738126] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.740296] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.743414] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.275253] Initialise system trusted keyrings [ 3.278074] Key type blacklist registered [ 3.279898] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.289613] zbud: loaded [ 3.292554] *** VALIDATE nfs *** [ 3.293937] *** VALIDATE nfs4 *** [ 3.295566] pstore: using deflate compression [ 3.299435] Platform Keyring initialized [ 3.413811] NET: Registered protocol family 38 [ 3.416211] Key type asymmetric registered [ 3.418181] Asymmetric key parser 'x509' registered [ 3.420317] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.423784] io scheduler mq-deadline registered [ 3.426968] io scheduler kyber registered [ 3.428971] io scheduler bfq registered [ 3.430856] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.434067] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.436609] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.440254] ACPI: Power Button [PWRF] [ 3.446418] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.454725] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.474374] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.493395] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.523231] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.555763] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.587948] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.593569] Non-volatile memory driver v1.3 [ 3.595983] Linux agpgart interface v0.103 [ 3.630474] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.634592] vda: detected capacity change from 0 to 76681216 [ 3.652837] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.656993] vdb: detected capacity change from 0 to 1073741824 [ 3.771112] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.773492] vdc: detected capacity change from 0 to 2621440000 [ 3.967985] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.022402] vdd: detected capacity change from 0 to 2621440000 [ 4.045399] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.049949] vde: detected capacity change from 0 to 4294967296 [ 4.081437] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.085702] vdf: detected capacity change from 0 to 4294967296 [ 4.095149] libphy: Fixed MDIO Bus: probed [ 4.100797] usbcore: registered new interface driver usbserial_generic [ 4.104770] usbserial: USB Serial support registered for generic [ 4.107961] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.113957] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.116296] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.119110] mousedev: PS/2 mouse device common for all mice [ 4.124079] rtc_cmos 00:05: RTC can wake from S4 [ 4.129402] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.132215] rtc_cmos 00:05: registered as rtc0 [ 4.138074] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.140855] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.142812] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.149338] intel_pstate: CPU model not supported [ 4.152372] hid: raw HID events driver (C) Jiri Kosina [ 4.154690] usbcore: registered new interface driver usbhid [ 4.156981] usbhid: USB HID core driver [ 4.158689] drop_monitor: Initializing network drop monitor service [ 4.161188] Initializing XFRM netlink socket [ 4.163401] NET: Registered protocol family 10 [ 4.166534] Segment Routing with IPv6 [ 4.168330] NET: Registered protocol family 17 [ 4.170459] mpls_gso: MPLS GSO support [ 4.175687] RAS: Correctable Errors collector initialized. [ 4.178123] AVX version of gcm_enc/dec engaged. [ 4.179677] AES CTR mode by8 optimization enabled [ 4.258279] sched_clock: Marking stable (4258209899, 0)->(5522732778, -1264522879) [ 4.261903] registered taskstats version 1 [ 4.264160] Loading compiled-in X.509 certificates [ 4.266642] zswap: loaded using pool lzo/zbud [ 4.268371] hpet1: lost 7 rtc interrupts [ 4.294981] Key type big_key registered [ 4.308546] Key type encrypted registered [ 4.310503] ima: No TPM chip found, activating TPM-bypass! [ 4.313941] ima: Allocated hash algorithm: sha1 [ 4.315968] ima: No architecture policies found [ 4.331487] evm: Initialising EVM extended attributes: [ 4.334604] evm: security.selinux [ 4.336590] evm: security.ima [ 4.339495] evm: security.capability [ 4.341784] evm: HMAC attrs: 0x1 [ 4.344984] rtc_cmos 00:05: setting system clock to 2026-09-03 19:16:03 UTC (1788462963) [ 4.355375] debug: unmapping init [mem 0xffffffffbd203000-0xffffffffbd3fffff] [ 4.359720] debug: unmapping init [mem 0xffffffffbbf82000-0xffffffffbc258fff] [ 4.372139] Write protecting the kernel read-only data: 28672k [ 4.411248] debug: unmapping init [mem 0xffffffffba603000-0xffffffffba7fffff] [ 4.417260] debug: unmapping init [mem 0xffffffffbaf14000-0xffffffffbaffffff] [ 4.490375] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.642985] systemd[1]: Detected virtualization kvm. [ 4.646117] systemd[1]: Detected architecture x86-64. [ 4.649722] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.680364] systemd[1]: No hostname configured. [ 4.682288] systemd[1]: Set hostname to . [ 4.684846] random: systemd: uninitialized urandom read (16 bytes read) [ 4.687405] systemd[1]: Initializing machine ID from random generator. [ 4.786986] random: ln: uninitialized urandom read (6 bytes read) [ 4.907953] random: systemd: uninitialized urandom read (16 bytes read) [ 4.912039] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.928182] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.944280] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.950267] device-mapper: uevent: version 1.0.3 [ 5.952852] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [[ 6.431188] random: fast init done  OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.510976] virtio_net virtio0 ens2: renamed from eth0 [ 7.751372] scsi host0: ata_piix [ 7.770730] scsi host1: ata_piix [ 7.772460] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.776579] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.896495] random: crng init done [ 12.912925] random: 7 urandom warning(s) missed due to ratelimiting [ 13.162543] dracut-initqueue[579]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.008136] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.771417] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.843264] SELinux: Disabled at runtime. [ 23.916247] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.925387] systemd[1]: Detected virtualization kvm. [ 23.928147] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.469967] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.479312] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.504162] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.523858] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.541252] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.567757] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.601067] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... [[ 27.475167] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 28.836362] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.017689] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 32.181844] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.156970] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.415826] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ *] A start job is running for Configur…only root support (14s / no limit)[ 40.898947] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (14s / no limit)[ 41.488079] NFS: Registering the id_resolver key type [ 41.495177] Key type id_resolver registered [ 41.508907] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (16s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg323-server login: [ 104.608626] libcfs: loading out-of-tree module taints kernel. [ 104.729709] Key type ._llcrypt registered [ 104.735179] Key type .llcrypt registered [ 104.908224] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 127.212608] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 129.306677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 129.317242] alg: No test for adler32 (adler32-zlib) [ 130.737741] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 131.646731] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 133.431161] Key type lgssc registered [ 135.222704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 152.713503] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 168.130566] hrtimer: interrupt took 2749034 ns [ 194.650343] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 208.695114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 208.813516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.204396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 210.246244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 210.343233] Lustre: lustre-MDT0000: new disk, initializing [ 210.417547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.436165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 215.217688] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.348230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 228.529930] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 228.572698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 228.581968] Lustre: Skipped 1 previous similar message [ 228.693114] Lustre: lustre-MDT0001: new disk, initializing [ 228.782664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.814622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 228.819390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 233.437746] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.958521] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 248.318289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.611422] Lustre: lustre-OST0000: new disk, initializing [ 248.618927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 248.626643] Lustre: 8449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.726237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.900652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 251.915370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 251.976732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 255.251575] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 268.799202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 268.959995] Lustre: lustre-OST0001: new disk, initializing [ 268.966157] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 268.976960] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 269.047616] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 274.791973] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.528836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 276.538834] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 276.591758] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 287.334404] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 295.233378] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 302.362892] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing check_logdir /tmp/testlogs/ [ 307.406764] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing yml_node [ 312.137873] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 314.748170] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 317.405678] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 319.314846] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 15:21:17 EDT 2026 [ 338.210662] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 339.915863] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 356.832842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 356.849169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.876155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.368358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.376509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.488154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 363.499662] Lustre: Skipped 6 previous similar messages [ 368.611935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 368.621817] Lustre: Skipped 3 previous similar messages [ 370.143745] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 370.301853] Lustre: server umount lustre-MDT0000 complete [ 373.962400] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463333 with bad export cookie 8388595894292558863 [ 373.971099] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 373.976040] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 378.854099] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.854248] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 378.857712] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 378.884525] Lustre: Skipped 3 previous similar messages [ 378.900140] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 380.401184] Lustre: server umount lustre-MDT0001 complete [ 384.031834] Lustre: server umount lustre-OST0000 complete [ 388.013649] Lustre: server umount lustre-OST0001 complete [ 396.065784] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 407.060895] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 444.204779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.455394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 444.495835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 444.604567] Lustre: lustre-MDT0000: new disk, initializing [ 444.730945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.753709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 449.479888] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 460.046345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.181672] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 460.276087] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 460.282682] Lustre: Skipped 1 previous similar message [ 460.367749] Lustre: lustre-MDT0001: new disk, initializing [ 460.486789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 460.538602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 460.553099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 464.877604] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 474.471823] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 481.311691] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 489.166068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 489.417774] Lustre: lustre-OST0000: new disk, initializing [ 489.423449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 489.433178] Lustre: 17045:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 489.566154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.349220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 491.356912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 491.496915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 496.072987] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 505.587784] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 510.908796] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 511.161847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 517.216684] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 517.484172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 522.223104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 522.224969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.236579] LustreError: Skipped 1 previous similar message [ 522.243108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 522.275965] Lustre: Skipped 1 previous similar message [ 522.317657] Lustre: Skipped 2 previous similar messages [ 527.001965] Lustre: server umount lustre-OST0000 complete [ 532.451991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 532.465281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.501873] Lustre: Skipped 1 previous similar message [ 536.724457] Lustre: server umount lustre-MDT0000 complete [ 537.577854] LustreError: 15087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.609841] LustreError: 15087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 539.968458] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463499 with bad export cookie 8388595894292560396 [ 539.973137] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 539.976727] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 540.429052] Lustre: server umount lustre-MDT0001 complete [ 544.953242] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 15:25:02 (1788463502) [ 547.442601] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 549.107893] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 15:25:07 (1788463507) [ 555.989664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.434828] LustreError: 19453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.497165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 561.646687] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.489467] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 566.753699] LustreError: 19453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.116212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.883765] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 585.305319] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 591.630714] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 600.216670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 606.462767] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 613.461921] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 623.863883] Lustre: server umount lustre-OST0000 complete [ 628.705634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.714632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.723921] Lustre: Skipped 1 previous similar message [ 628.739433] Lustre: Skipped 3 previous similar messages [ 636.978584] Lustre: server umount lustre-MDT0000 complete [ 638.959824] LustreError: 19453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.993625] LustreError: 19453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 641.306062] LustreError: 22234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463600 with bad export cookie 8388595894292561278 [ 641.315751] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 641.320685] LustreError: 22234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 651.985398] Lustre: server umount lustre-MDT0001 complete [ 661.859442] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 664.534660] Key type lgssc unregistered [ 664.889604] LNet: 22988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.908655] LNetError: 22988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.928815] LNet: Removed LNI 192.168.203.123@tcp [ 665.689580] Key type .llcrypt unregistered [ 665.693449] Key type ._llcrypt unregistered [ 688.074786] Key type ._llcrypt registered [ 688.083229] Key type .llcrypt registered [ 688.226490] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 689.256259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 689.298992] alg: No test for adler32 (adler32-zlib) [ 690.425789] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 690.694693] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 692.431171] Key type lgssc registered [ 693.836975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 702.099508] loop: module loaded [ 702.179208] loop0: detected capacity change from 0 to 204800000 [ 702.429378] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 702.441133] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 702.841950] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 708.923863] loop0: detected capacity change from 0 to 204800000 [ 709.064994] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.085887] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.125479] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 714.596699] loop0: detected capacity change from 0 to 204800000 [ 719.783675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 719.802288] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.019352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 721.054775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 721.119212] Lustre: lustre-MDT0000: new disk, initializing [ 721.163074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 721.176703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 725.353130] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 733.614619] loop1: detected capacity change from 0 to 204800000 [ 739.171995] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 739.413783] Lustre: lustre-OST07c6: new disk, initializing [ 739.417850] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 739.422802] Lustre: Skipped 1 previous similar message [ 739.427940] Lustre: 26173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 739.497932] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 744.573960] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 746.040959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 746.053851] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 746.103498] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 750.109043] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 750.123955] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 750.132078] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 750.137290] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 750.143696] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 750.149751] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 761.957355] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 761.967109] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 761.976196] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 761.986479] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 761.995704] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 762.013137] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 767.279891] Lustre: 25160:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 767.296706] Lustre: 25160:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 767.305030] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 767.311285] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 767.324478] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 767.329786] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 767.340704] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 767.349237] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 767.358844] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 767.366283] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 767.377509] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 767.387603] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.929548] Lustre: 26856:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 772.943496] Lustre: 26856:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 772.957333] Lustre: 26856:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 772.974081] Lustre: 26856:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.987715] Lustre: 26856:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 773.006116] Lustre: 26856:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.020158] Lustre: 26856:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 773.047419] Lustre: 26856:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.054455] Lustre: 26856:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 773.066439] Lustre: 26856:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.071868] Lustre: 26856:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 773.078151] Lustre: 26856:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.023080] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 781.056123] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 781.070417] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 781.086295] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 781.109446] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 781.134841] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 781.166511] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 781.190181] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 781.232175] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 781.243154] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 781.261225] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 781.280932] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 792.960910] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 792.970021] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 792.976596] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 792.982569] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.988147] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 792.993140] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 793.002138] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 793.021119] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 793.030101] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 793.039432] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 793.051399] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 793.062058] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 838.115171] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.133915] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 838.472881] Lustre: server umount lustre-OST07c6 complete [ 846.779795] Lustre: server umount lustre-MDT0000 complete [ 862.952910] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 865.669788] Key type lgssc unregistered [ 865.951273] LNet: 28726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.967912] LNetError: 28726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.993935] LNet: Removed LNI 192.168.203.123@tcp [ 866.866369] Key type .llcrypt unregistered [ 866.869301] Key type ._llcrypt unregistered [ 884.833273] Key type ._llcrypt registered [ 884.835967] Key type .llcrypt registered [ 884.947303] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 886.063074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 886.103524] alg: No test for adler32 (adler32-zlib) [ 887.084617] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 887.301156] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 888.951192] Key type lgssc registered [ 889.911310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 896.720936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 896.747954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 898.170342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 902.925898] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 908.782099] LustreError: 29993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 908.800824] LustreError: 29993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 910.990851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.445186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 916.060763] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 923.604510] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 928.794263] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 935.281559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 935.781513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 941.309627] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 942.963234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 947.928106] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 962.529316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 962.538659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.558572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 963.555919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 963.557502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 963.568763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 967.438171] Lustre: server umount lustre-OST0000 complete [ 972.776155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 972.799219] LustreError: Skipped 1 previous similar message [ 972.803742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 972.816962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 972.820489] Lustre: Skipped 1 previous similar message [ 977.327353] Lustre: server umount lustre-MDT0000 complete [ 978.934299] LustreError: 30721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 978.972644] LustreError: 30721:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 980.690845] LustreError: 29971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463939 with bad export cookie 12563433575921533968 [ 980.702636] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.711788] LustreError: 29971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 980.981220] Lustre: server umount lustre-MDT0001 complete [ 990.880706] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 993.126891] Key type lgssc unregistered [ 993.401968] LNet: 33728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.409711] LNetError: 33728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.447887] LNet: Removed LNI 192.168.203.123@tcp [ 994.227287] Key type .llcrypt unregistered [ 994.229727] Key type ._llcrypt unregistered [ 1015.849453] Key type ._llcrypt registered [ 1015.854392] Key type .llcrypt registered [ 1016.099085] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1018.298807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1018.329309] alg: No test for adler32 (adler32-zlib) [ 1019.449117] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 1019.693509] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1021.451507] Key type lgssc registered [ 1022.662450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1030.093215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1030.132737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.655482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.453318] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1040.874630] LustreError: 35002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.910959] LustreError: 35002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1045.209358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1045.617943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1050.106472] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1057.012338] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1062.849857] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1069.405819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1069.859670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.038768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1076.850605] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1084.741444] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1096.680961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1096.686453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1096.711857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1096.739410] Lustre: Skipped 1 previous similar message [ 1101.801167] Lustre: server umount lustre-OST0000 complete [ 1106.913338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.914728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1106.954129] Lustre: Skipped 1 previous similar message [ 1106.956327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1112.040786] Lustre: server umount lustre-MDT0000 complete [ 1112.041175] LustreError: 35002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1112.083934] LustreError: 35002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1115.777205] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464074 with bad export cookie 159292788251119271 [ 1115.785295] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1115.793453] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1116.057197] Lustre: server umount lustre-MDT0001 complete [ 1126.506619] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1129.338690] Key type lgssc unregistered [ 1129.648886] LNet: 38536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.659735] LNetError: 38536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.689933] LNet: Removed LNI 192.168.203.123@tcp [ 1130.495480] Key type .llcrypt unregistered [ 1130.499144] Key type ._llcrypt unregistered [ 1154.914050] Key type ._llcrypt registered [ 1154.915584] Key type .llcrypt registered [ 1155.055350] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1156.073657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1156.234996] alg: No test for adler32 (adler32-zlib) [ 1157.387082] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 1157.649063] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1159.319188] Key type lgssc registered [ 1160.574295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1167.820980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1167.840518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1169.245544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.788691] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1179.631817] LustreError: 39949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1179.647232] LustreError: 39949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1182.343078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1182.692941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1187.128159] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1194.231906] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1199.450747] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1205.475896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1205.747601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1211.258211] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1213.933894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1217.989159] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1224.731169] Lustre: server umount lustre-MDT0000 complete [ 1225.594076] LustreError: 39944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1228.097970] LustreError: 39930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464187 with bad export cookie 16066025162568456000 [ 1228.104140] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1228.107545] LustreError: 39930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1229.283676] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.302677] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1229.303783] LustreError: 39943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.728551] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 1233.314761] LustreError: 39944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1233.319796] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1233.332378] LustreError: 39944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1234.485737] Lustre: server umount lustre-MDT0001 complete [ 1256.269359] Lustre: server umount lustre-OST0000 complete [ 1273.672624] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1277.304078] Key type lgssc unregistered [ 1277.671957] LNet: 43807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1277.689609] LNetError: 43807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.709621] LNet: Removed LNI 192.168.203.123@tcp [ 1278.712291] Key type .llcrypt unregistered [ 1278.715960] Key type ._llcrypt unregistered [ 1302.804157] Key type ._llcrypt registered [ 1302.807659] Key type .llcrypt registered [ 1302.920771] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1304.156918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1304.340256] alg: No test for adler32 (adler32-zlib) [ 1305.428960] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 1305.686114] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1307.487342] Key type lgssc registered [ 1308.698677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1316.573910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1316.585361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1318.129212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.661271] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1328.610341] LustreError: 45222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.641311] LustreError: 45222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1331.223979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.533954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1335.978195] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1343.433996] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1349.417782] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1355.669509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1356.043187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1362.862600] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1363.198981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1371.304121] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1378.796084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1378.810712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1378.822962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1378.843078] Lustre: Skipped 1 previous similar message [ 1383.905302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1383.917242] Lustre: Skipped 1 previous similar message [ 1384.263708] Lustre: server umount lustre-OST0000 complete [ 1403.361111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1403.373301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1403.384237] Lustre: Skipped 1 previous similar message [ 1403.389639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1403.396688] Lustre: Skipped 1 previous similar message [ 1404.391785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1408.628454] Lustre: server umount lustre-MDT0000 complete [ 1409.518197] LustreError: 45217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1409.543734] LustreError: 45217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1410.544264] LustreError: 45951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1412.740290] LustreError: 45203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464371 with bad export cookie 11064509467537169564 [ 1412.752322] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1412.753524] LustreError: 45203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1419.165622] Lustre: server umount lustre-MDT0001 complete [ 1432.026329] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1434.941463] Key type lgssc unregistered [ 1435.174955] LNet: 48935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.179290] LNetError: 48935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1435.208586] LNet: Removed LNI 192.168.203.123@tcp [ 1435.994867] Key type .llcrypt unregistered [ 1435.996831] Key type ._llcrypt unregistered [ 1460.029992] Key type ._llcrypt registered [ 1460.032703] Key type .llcrypt registered [ 1460.140576] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1461.490774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1461.526238] alg: No test for adler32 (adler32-zlib) [ 1462.638423] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 1462.843989] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1464.529724] Key type lgssc registered [ 1465.736704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1473.322221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1473.368751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1474.884308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.547431] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1485.293149] LustreError: 50348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1485.303639] LustreError: 50348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1490.402402] LustreError: 50347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1491.828614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1492.181659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1498.603677] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1505.533160] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1510.620168] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1516.729554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1517.028465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1520.117462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1522.676506] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1530.358657] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1535.362542] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1538.567982] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1550.477340] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0315h41m48s [ 1551.898532] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1553.890125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1553.897391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1553.915848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1554.993424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1555.010073] Lustre: Skipped 4 previous similar messages [ 1557.372660] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 1559.946658] Lustre: server umount lustre-MDT0000 complete [ 1561.060048] LustreError: 50348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1561.074668] LustreError: 50348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1562.990237] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1563.496605] LustreError: 50344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1569.329477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.529262] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1569.636582] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (not set up) [ 1569.650426] Lustre: Skipped 1 previous similar message [ 1569.886241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1569.966523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1570.948681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1573.600578] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1576.629166] Lustre: DEBUG MARKER: Wait for df (36658) ... [ 1577.792880] Lustre: DEBUG MARKER: done [ 1581.541506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1581.543505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1581.567395] Lustre: Skipped 3 previous similar messages [ 1581.573272] Lustre: Skipped 1 previous similar message [ 1591.782142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1591.788064] Lustre: Skipped 2 previous similar messages [ 1595.359157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1595.662433] Lustre: server umount lustre-OST0000 complete [ 1602.019255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1602.019516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1609.187639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1609.196526] Lustre: Skipped 5 previous similar messages [ 1614.304732] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1614.470900] Lustre: server umount lustre-MDT0000 complete [ 1615.332711] LustreError: 51076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1615.351316] LustreError: 51076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1618.002255] LustreError: 52972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464577 with bad export cookie 14421778615633815759 [ 1618.012157] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1618.012916] LustreError: 52972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1618.362865] Lustre: server umount lustre-MDT0001 complete [ 1628.101326] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1630.552592] Key type lgssc unregistered [ 1630.919713] LNet: 55271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1630.925718] LNetError: 55271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1630.944818] LNet: Removed LNI 192.168.203.123@tcp [ 1631.758877] Key type .llcrypt unregistered [ 1631.764390] Key type ._llcrypt unregistered [ 1669.821447] Key type ._llcrypt registered [ 1669.823435] Key type .llcrypt registered [ 1669.904544] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1671.202720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1671.238428] alg: No test for adler32 (adler32-zlib) [ 1672.283253] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 1672.543627] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1674.255159] Key type lgssc registered [ 1675.564107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1682.740669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1682.761712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.014564] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1684.032719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1684.164792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.105783] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1695.197103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1695.274918] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1695.314902] Lustre: 57474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1695.483852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1700.607699] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1707.687134] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1715.678564] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1724.987092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1725.321067] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1725.848543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1731.677727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1732.788871] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1740.707867] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1748.505509] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0315h45m06s [ 1750.378928] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1757.543622] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1759.541805] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1759.723277] LustreError: 58937:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1780.096842] Lustre: lustre-MDT0000: Client c7dc1ec8-cbae-4b36-a6d0-cf68cba3e007 (at 192.168.203.23@tcp) reconnecting [ 1799.743128] LustreError: 58937:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1801.675872] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1803.818285] Lustre: DEBUG MARKER: done [ 1813.475422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1813.479553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1813.495537] Lustre: Skipped 1 previous similar message [ 1813.515639] Lustre: Skipped 1 previous similar message [ 1815.316548] Lustre: server umount lustre-OST0000 complete [ 1818.593714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1818.594884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.598123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1818.625539] Lustre: Skipped 1 previous similar message [ 1819.680553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1819.686311] Lustre: Skipped 2 previous similar messages [ 1824.712345] Lustre: server umount lustre-MDT0000 complete [ 1827.729206] LustreError: 57469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464786 with bad export cookie 2163668226770000969 [ 1827.730877] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1827.745763] LustreError: 57469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1827.997372] Lustre: server umount lustre-MDT0001 complete [ 1838.465102] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1841.209894] Key type lgssc unregistered [ 1841.576098] LNet: 61803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.584183] LNetError: 61803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.597869] LNet: Removed LNI 192.168.203.123@tcp [ 1842.697235] Key type .llcrypt unregistered [ 1842.699239] Key type ._llcrypt unregistered [ 1885.273760] Key type ._llcrypt registered [ 1885.275481] Key type .llcrypt registered [ 1885.346528] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1885.969380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1886.093513] alg: No test for adler32 (adler32-zlib) [ 1887.164889] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 1887.416279] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1889.111219] Key type lgssc registered [ 1890.340589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1892.858724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1892.911386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.221762] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1894.231669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1894.351088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.378179] Lustre: Failing over lustre-MDT0000 [ 1895.582178] Lustre: server umount lustre-MDT0000 complete [ 1902.842590] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 15:47:40 (1788464860) [ 1911.842157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.180803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1916.904660] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1925.325548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.465922] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1925.504304] Lustre: 64597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1925.797670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1930.487451] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1937.433043] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1942.473419] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1948.467143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1948.626817] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1948.833235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.206923] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1955.977327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1962.721922] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1968.946218] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1974.938086] Lustre: server umount lustre-MDT0000 complete [ 1976.804355] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1976.814636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1978.151471] LustreError: 66452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464937 with bad export cookie 9039286754165437654 [ 1978.157711] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1978.166618] LustreError: 66452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1978.447129] Lustre: server umount lustre-MDT0001 complete [ 1981.304719] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1985.720241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1995.560832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.303234] Lustre: 63521:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788464941/real 1788464941] req@0000000009664e71 x1875341314397440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788464957 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1998.330992] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.341180] Lustre: Skipped 2 previous similar messages [ 2000.415552] Lustre: 63521:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788464943/real 1788464943] req@000000007013ac2e x1875341314397568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788464959 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2003.423487] Lustre: 63519:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788464946/real 1788464946] req@0000000047c30d94 x1875341314397696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788464962 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2003.487777] LustreError: 63517:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ec9f6f6d x1875341314398592/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2003.926730] LustreError: 66445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.966591] LustreError: 66445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2004.084980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.273588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2008.162037] LustreError: 68367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.175758] LustreError: 68367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2009.134356] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2010.083875] Lustre: 63521:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788464952/real 1788464952] req@00000000657c7a6d x1875341314398080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788464968 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2010.119903] Lustre: 63521:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2013.672163] LustreError: 68366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2013.690442] LustreError: 68366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2017.521392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.768396] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2017.773339] Lustre: Skipped 1 previous similar message [ 2017.780374] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2017.965394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2022.258448] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2023.410339] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2029.680877] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2034.862699] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2043.872918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2043.875539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.895159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2043.899419] Lustre: Skipped 1 previous similar message [ 2047.954938] Lustre: server umount lustre-MDT0000 complete [ 2048.994462] LustreError: 69421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2049.022027] LustreError: 69421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2053.003077] LustreError: 66452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465012 with bad export cookie 9039286754165439929 [ 2053.031090] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2053.037558] LustreError: 66452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2053.439642] Lustre: server umount lustre-MDT0001 complete [ 2061.785562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2070.432720] Lustre: 63520:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788465013/real 1788465013] req@0000000025d92530 x1875341314424064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788465029 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2070.477741] Lustre: 63520:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2070.489150] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2070.501333] Lustre: Skipped 2 previous similar messages [ 2071.508307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.691081] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7d71ff6924ecc31c [ 2078.705168] Lustre: MGC192.168.203.123@tcp: Connection restored to 0@lo (at 0@lo) [ 2079.022412] LustreError: 66446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.126822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.198878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2080.097168] Lustre: 63520:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788465022/real 1788465022] req@000000000971d93d x1875341314424576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788465038 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2080.129814] Lustre: 63520:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2083.816254] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2090.477889] LustreError: 69421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.506392] LustreError: 69421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2091.829351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.598110] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2097.125946] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2101.941342] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2107.913742] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2117.602284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2117.615676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.637474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2119.656506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2122.737747] Lustre: server umount lustre-MDT0000 complete [ 2124.768475] LustreError: 71053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.792556] LustreError: 71053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2127.222274] LustreError: 70197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465086 with bad export cookie 9039286754165441308 [ 2127.231307] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2127.240953] LustreError: 70197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2127.655811] Lustre: server umount lustre-MDT0001 complete [ 2134.146399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2136.746498] Lustre: DEBUG MARKER: files compared the same [ 2148.102152] Lustre: server umount lustre-OST0000 complete [ 2163.189498] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2166.652277] Key type lgssc unregistered [ 2167.044674] LNet: 74329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2167.057486] LNetError: 74329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2167.079545] LNet: Removed LNI 192.168.203.123@tcp [ 2167.996923] Key type .llcrypt unregistered [ 2167.999785] Key type ._llcrypt unregistered [ 2193.038455] Key type ._llcrypt registered [ 2193.042233] Key type .llcrypt registered [ 2193.148500] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2193.880868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2193.908423] alg: No test for adler32 (adler32-zlib) [ 2194.985820] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 2195.231056] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2196.943663] Key type lgssc registered [ 2197.732618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2204.316303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2204.327822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.674397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.945473] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug +malloc all [ 2214.881092] LustreError: 75741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.897601] LustreError: 75741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2220.004310] LustreError: 75740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2220.060726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.297818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2226.360991] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug +malloc all [ 2235.104640] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2240.668229] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2247.143279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2247.395112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.487756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2253.630438] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug +malloc all [ 2262.320187] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2276.320743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2276.323585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2276.326303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2276.332827] LustreError: Skipped 1 previous similar message [ 2276.357870] Lustre: Skipped 1 previous similar message [ 2281.449086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2281.459190] Lustre: Skipped 1 previous similar message [ 2281.762993] Lustre: server umount lustre-OST0000 complete [ 2286.560321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2286.590476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2286.604439] Lustre: Skipped 2 previous similar messages [ 2291.681136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2291.686191] Lustre: Skipped 1 previous similar message [ 2296.799912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2296.807207] Lustre: Skipped 1 previous similar message [ 2299.871170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2300.188542] Lustre: server umount lustre-MDT0000 complete [ 2301.925660] LustreError: 75741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.944624] LustreError: 75741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2303.704720] LustreError: 75721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465262 with bad export cookie 3344825187367271093 [ 2303.712137] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2303.731177] LustreError: 75721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2304.085957] Lustre: server umount lustre-MDT0001 complete [ 2314.748607] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2317.707304] Key type lgssc unregistered [ 2318.019180] LNet: 79295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.033298] LNetError: 79295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2319.082187] LNet: Removed LNI 192.168.203.123@tcp [ 2319.825462] Key type .llcrypt unregistered [ 2319.829297] Key type ._llcrypt unregistered [ 2343.287837] Key type ._llcrypt registered [ 2343.290193] Key type .llcrypt registered [ 2343.388118] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2344.155328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2344.267190] alg: No test for adler32 (adler32-zlib) [ 2345.390760] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 2345.641042] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2347.327172] Key type lgssc registered [ 2348.527386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2356.321626] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2356.350441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2372.000472] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2377.119335] LustreError: 80683:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 4: rc = -110 [ 2403.743190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2409.344901] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2416.455218] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2422.927726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.125394] Lustre: *** cfs_fail_loc=706, val=0*** [ 2423.530752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.643671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2428.057654] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2428.896744] LustreError: 80708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2428.926646] LustreError: 80708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2434.022855] LustreError: 80709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2436.469258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2436.891635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2441.801480] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2449.409563] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2454.641669] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2457.569203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2457.588076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.607270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2458.596891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.602835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2458.608166] Lustre: Skipped 1 previous similar message [ 2462.705857] Lustre: server umount lustre-OST0000 complete [ 2467.808542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2467.821436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.841836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2473.270794] Lustre: server umount lustre-MDT0000 complete [ 2473.955988] LustreError: 82554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2473.979534] LustreError: 82554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2477.679028] LustreError: 80689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465436 with bad export cookie 13859396676906040358 [ 2477.695790] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2477.702108] LustreError: 80689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2478.155317] Lustre: server umount lustre-MDT0001 complete [ 2488.967780] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2491.749733] Key type lgssc unregistered [ 2492.099795] LNet: 84295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2492.108578] LNetError: 84295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2492.132698] LNet: Removed LNI 192.168.203.123@tcp [ 2493.046966] Key type .llcrypt unregistered [ 2493.049770] Key type ._llcrypt unregistered [ 2520.962263] Key type ._llcrypt registered [ 2520.964908] Key type .llcrypt registered [ 2521.171640] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2522.639099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2522.785712] alg: No test for adler32 (adler32-zlib) [ 2524.182685] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 2524.508130] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2526.239264] Key type lgssc registered [ 2527.356329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2535.684455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2535.724408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2541.550089] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2548.752030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2549.348673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2552.774467] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2554.336289] LustreError: 86395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.343540] LustreError: 86395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2559.458672] LustreError: 86398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.483700] LustreError: 86398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2561.701050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2562.264711] LustreError: 86399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.288897] LustreError: 86399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2562.305928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2567.659156] LustreError: 86395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.683559] LustreError: 86395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2569.539339] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2572.772293] LustreError: 86398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.782013] LustreError: 86398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2580.411381] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2580.420754] Lustre: MGS: Client e55d43d9-97d9-4459-b239-fd4e1437d4d7 (at 0@lo) reconnecting [ 2580.452474] LustreError: 86409:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2580.690197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2580.750599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2585.299344] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2601.441901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2601.445754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.451471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2601.538741] Lustre: Skipped 1 previous similar message [ 2606.564533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2606.575541] Lustre: Skipped 1 previous similar message [ 2606.906251] Lustre: server umount lustre-OST0000 complete [ 2611.684353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2611.696263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.704223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2611.704231] Lustre: Skipped 1 previous similar message [ 2611.735166] Lustre: Skipped 1 previous similar message [ 2616.801550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2616.811290] Lustre: Skipped 2 previous similar messages [ 2621.921560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2624.991233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2625.284934] Lustre: server umount lustre-MDT0000 complete [ 2627.042921] LustreError: 86398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.083852] LustreError: 86398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2630.332988] Lustre: server umount lustre-MDT0001 complete [ 2634.669891] Lustre: server umount MGS complete [ 2644.842231] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:00:02 (1788465602) [ 2672.522427] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 2683.632838] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2716.529112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.072573] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2727.311566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2727.551318] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2727.590723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2727.676944] Lustre: lustre-MDT0001: new disk, initializing [ 2727.763387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2727.788026] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2727.798939] Lustre: 93598:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2727.810083] LustreError: 93598:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2731.275357] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2740.671271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2740.983788] Lustre: lustre-OST0000: new disk, initializing [ 2740.993299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2741.004453] Lustre: 94364:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2741.143234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2746.637469] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2747.936744] Lustre: 94738:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000849b8ba2 x1875341982344320/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788465723 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2755.124530] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2761.368574] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2761.387725] Lustre: MGS: Client 3a2aae64-e283-4751-ad1a-a31ffa6a28bc (at 0@lo) reconnecting [ 2761.451386] Lustre: 92864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2761.472857] LustreError: 92864:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2761.488217] LustreError: 92864:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2761.514355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2761.586358] Lustre: lustre-MDT0000: new disk, initializing [ 2761.703899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2761.734136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2761.885307] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2762.796044] Lustre: 95433:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005c1ee7b4 x1875341982350848/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788465737 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2766.573686] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2767.992890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2768.007971] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2768.039913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2771.959957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2771.974389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2799.081701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2799.083841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.088765] LustreError: Skipped 1 previous similar message [ 2799.114888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2799.139850] Lustre: Skipped 2 previous similar messages [ 2801.493384] Lustre: server umount lustre-OST0000 complete [ 2809.312957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2809.314919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2809.346590] Lustre: Skipped 2 previous similar messages [ 2811.016422] Lustre: server umount lustre-MDT0000 complete [ 2820.338870] Lustre: server umount lustre-MDT0001 complete [ 2823.169272] Lustre: server umount MGS complete [ 2832.311250] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:03:10 (1788465790) [ 2849.454332] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2852.309733] Key type lgssc unregistered [ 2852.551946] LNet: 97931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2852.559096] LNetError: 97931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2852.583099] LNet: Removed LNI 192.168.203.123@tcp [ 2853.281225] Key type .llcrypt unregistered [ 2853.284862] Key type ._llcrypt unregistered [ 2866.159198] Key type ._llcrypt registered [ 2866.160758] Key type .llcrypt registered [ 2866.311274] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2866.907939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2867.011238] alg: No test for adler32 (adler32-zlib) [ 2868.143368] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 2868.405383] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2870.128037] Key type lgssc registered [ 2871.049459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2882.329052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2882.350609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2883.992399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.700231] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2894.310238] LustreError: 99130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.340817] LustreError: 99130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2899.424831] LustreError: 99131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2900.791852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2901.101566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2906.824904] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2915.533989] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2922.289958] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2929.225987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2929.639973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2936.945421] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2946.301718] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2954.262889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2964.459252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2964.461336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.474479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2964.508574] Lustre: Skipped 1 previous similar message [ 2969.464752] Lustre: server umount lustre-OST0000 complete [ 2974.689769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.690715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.690720] Lustre: Skipped 1 previous similar message [ 2974.705677] Lustre: Skipped 1 previous similar message [ 2979.278567] Lustre: server umount lustre-MDT0000 complete [ 2979.813552] LustreError: 99131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2979.831417] LustreError: 99131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2982.730851] LustreError: 100973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465941 with bad export cookie 12509967023519696656 [ 2982.743844] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2982.745776] LustreError: 100973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2983.254317] Lustre: server umount lustre-MDT0001 complete [ 2993.771290] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2998.005929] Key type lgssc unregistered [ 2998.636791] LNet: 102685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2998.658286] LNetError: 102685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2999.717607] LNet: Removed LNI 192.168.203.123@tcp [ 3001.670527] Key type .llcrypt unregistered [ 3001.673609] Key type ._llcrypt unregistered [ 3019.647847] Key type ._llcrypt registered [ 3019.656117] Key type .llcrypt registered [ 3019.837198] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3020.769287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3020.811466] alg: No test for adler32 (adler32-zlib) [ 3021.896926] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 3022.152433] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3023.879642] Key type lgssc registered [ 3025.292508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3031.148852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3031.162478] LustreError: 103766:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3031.164355] Lustre: Skipped 1 previous similar message [ 3031.164370] LustreError: 103763:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3031.188329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.190808] LustreError: 103766:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3031.213346] LustreError: 103766:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3031.231928] LustreError: 103766:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3032.883329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3040.084271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.255366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3045.707900] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3051.254644] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing lsmod [ 3054.602770] LustreError: 104970:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3054.624163] LustreError: 104973:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3054.628904] LustreError: 104970:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4983 [ 3054.636142] LustreError: 104970:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3054.643629] LustreError: 104970:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3054.682948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3055.002404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3057.081645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3060.707310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3060.716492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3060.730369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3062.242288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3062.242728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3062.248411] Lustre: Skipped 1 previous similar message [ 3062.253780] Lustre: Skipped 2 previous similar messages [ 3063.970157] Lustre: server umount lustre-MDT0000 complete [ 3065.317328] LustreError: 103804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.349966] LustreError: 103804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3067.961479] LustreError: 103777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466027 with bad export cookie 4244076679047791956 [ 3067.977056] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3067.978301] LustreError: 103777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3069.471273] LustreError: 104234:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002747fe26 x1875342504514688/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3069.730710] Lustre: server umount lustre-MDT0001 complete [ 3085.098548] Lustre: server umount lustre-OST0000 complete [ 3096.168106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3096.742831] LustreError: 106036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3096.828833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3101.313225] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3102.177271] LustreError: 106037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3106.278391] LustreError: 106036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.805907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.352744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3116.481856] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3126.110390] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3132.180492] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3138.641050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3138.865916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.925213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:97) [ 3144.467559] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3152.095879] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3165.154678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3165.170573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.199220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3169.844743] Lustre: server umount lustre-OST0000 complete [ 3175.905033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3175.908119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3175.931600] Lustre: Skipped 2 previous similar messages [ 3175.953523] Lustre: Skipped 2 previous similar messages [ 3179.404823] Lustre: server umount lustre-MDT0000 complete [ 3181.024342] LustreError: 106031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3181.069242] LustreError: 106031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3182.592832] LustreError: 106018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466141 with bad export cookie 4244076679047792887 [ 3182.601767] LustreError: 106018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3182.607100] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3182.994287] Lustre: server umount lustre-MDT0001 complete [ 3192.332417] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3195.017826] Key type lgssc unregistered [ 3195.341398] LNet: 109571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3195.363230] LNetError: 109571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3195.378803] LNet: Removed LNI 192.168.203.123@tcp [ 3196.240187] Key type .llcrypt unregistered [ 3196.242379] Key type ._llcrypt unregistered [ 3218.246265] Key type ._llcrypt registered [ 3218.249356] Key type .llcrypt registered [ 3218.373196] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3218.978327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3219.043109] alg: No test for adler32 (adler32-zlib) [ 3220.186357] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 3220.513683] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3222.151164] Key type lgssc registered [ 3223.693052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3231.740962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3231.767803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3233.186642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3238.479374] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3243.502662] LustreError: 110990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3243.540506] LustreError: 110990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3246.186783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.466556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3250.053429] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3256.594649] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3261.107543] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3266.454801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3267.020235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3272.566076] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3278.324980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3280.297141] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3285.914458] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3289.407146] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3293.672147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3293.675389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3293.694914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3298.788764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3298.800931] Lustre: Skipped 2 previous similar messages [ 3299.513498] Lustre: server umount lustre-OST0000 complete [ 3302.880591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3302.893154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.910482] Lustre: Skipped 1 previous similar message [ 3302.918687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3303.920118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.024856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.037063] Lustre: Skipped 3 previous similar messages [ 3314.145442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3314.155040] Lustre: Skipped 1 previous similar message [ 3317.216574] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3317.474569] Lustre: server umount lustre-MDT0000 complete [ 3319.280934] LustreError: 110991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.325290] LustreError: 110991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3321.335154] LustreError: 111724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466280 with bad export cookie 1163656447838426435 [ 3321.345547] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3321.358104] LustreError: 111724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3321.694329] Lustre: server umount lustre-MDT0001 complete [ 3333.926700] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3336.692404] Key type lgssc unregistered [ 3337.028548] LNet: 114630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3337.042101] LNetError: 114630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3338.090562] LNet: Removed LNI 192.168.203.123@tcp [ 3338.982832] Key type .llcrypt unregistered [ 3338.984959] Key type ._llcrypt unregistered [ 3354.826453] Key type ._llcrypt registered [ 3354.828577] Key type .llcrypt registered [ 3354.916165] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3356.034991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3356.044786] alg: No test for adler32 (adler32-zlib) [ 3357.149367] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 3357.415831] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3359.087151] Key type lgssc registered [ 3360.187237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3367.371563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3367.394840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.893862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.247768] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3378.971980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3379.239830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3382.212388] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3388.028726] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3394.019065] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3401.107646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3401.413756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3406.830921] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3414.212786] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3415.724361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3427.811739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3427.818055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.847419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3430.371482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3430.383563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3430.406122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3433.478627] Lustre: server umount lustre-OST0000 complete [ 3440.613114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3440.622853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.642058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3440.652836] Lustre: Skipped 2 previous similar messages [ 3446.757552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3446.761922] Lustre: Skipped 2 previous similar messages [ 3451.361562] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3451.564584] Lustre: server umount lustre-MDT0000 complete [ 3451.880458] LustreError: 116653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.893142] LustreError: 116653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3455.257663] LustreError: 115828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466414 with bad export cookie 18189128419817355393 [ 3455.264942] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3455.270603] LustreError: 115828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3455.564882] Lustre: server umount lustre-MDT0001 complete [ 3466.588703] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3469.423882] Key type lgssc unregistered [ 3469.719795] LNet: 119398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3469.727436] LNetError: 119398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3469.755272] LNet: Removed LNI 192.168.203.123@tcp [ 3470.501559] Key type .llcrypt unregistered [ 3470.503988] Key type ._llcrypt unregistered [ 3492.739306] Key type ._llcrypt registered [ 3492.741505] Key type .llcrypt registered [ 3492.842681] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3493.970204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3494.013403] alg: No test for adler32 (adler32-zlib) [ 3495.130808] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 3495.369326] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3497.031153] Key type lgssc registered [ 3498.330509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3505.675799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3505.694330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3507.136881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3511.681035] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3517.421760] LustreError: 120820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3517.448076] LustreError: 120820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3520.257124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.600252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3525.158927] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3532.461539] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3538.035602] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3544.583636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3544.976698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3551.520613] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3552.188488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 3559.232582] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3565.324308] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3568.683377] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3568.689803] Lustre: Skipped 2 previous similar messages [ 3571.760200] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3571.799852] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3572.504059] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3572.509804] Lustre: Skipped 2 previous similar messages [ 3577.418141] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3577.422576] Lustre: Skipped 1 previous similar message [ 3582.812352] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.23@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3582.820661] Lustre: Skipped 2 previous similar messages [ 3583.057381] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.23@tcp 192.168.0.[2,10]@tcp [ 3590.372935] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.23/24@tcp in log lustre-MDT0000 [ 3590.382158] Lustre: Skipped 2 previous similar messages [ 3593.063513] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.23/24@tcp [ 3593.071015] Lustre: Skipped 1 previous similar message [ 3600.877309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3600.886378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3600.890478] LustreError: Skipped 1 previous similar message [ 3600.917972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3601.493125] Lustre: server umount lustre-OST0000 complete [ 3607.528602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3607.535233] LustreError: Skipped 3 previous similar messages [ 3607.537598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3607.555574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3608.553292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3608.566158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3608.571838] Lustre: Skipped 1 previous similar message [ 3613.667424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3613.687101] Lustre: Skipped 1 previous similar message [ 3618.719141] Lustre: 120129:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788466561/real 1788466561] req@00000000a5854972 x1875343000533248/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788466577 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3618.750376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3618.768540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3618.781956] Lustre: Skipped 1 previous similar message [ 3619.295185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3619.609885] Lustre: server umount lustre-MDT0000 complete [ 3619.812819] Lustre: 120132:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788466562/real 1788466562] req@000000007ed87e39 x1875343000534016/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788466578 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3623.156275] LustreError: 123449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466582 with bad export cookie 1998030447409519171 [ 3623.157223] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3623.170622] LustreError: 123449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3623.598046] Lustre: server umount lustre-MDT0001 complete [ 3633.892778] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3636.849622] Key type lgssc unregistered [ 3637.129731] LNet: 125618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3637.145642] LNetError: 125618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3637.167957] LNet: Removed LNI 192.168.203.123@tcp [ 3637.893204] Key type .llcrypt unregistered [ 3637.895681] Key type ._llcrypt unregistered [ 3664.133165] Key type ._llcrypt registered [ 3664.135727] Key type .llcrypt registered [ 3664.226864] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3665.380871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3665.435606] alg: No test for adler32 (adler32-zlib) [ 3666.593318] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 3666.843542] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3668.567171] Key type lgssc registered [ 3669.727329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3677.256675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3677.277407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.748269] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3678.778932] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.23/24@tcp [ 3678.810908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3683.117578] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3687.907849] LustreError: 127176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3687.943158] LustreError: 127176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3691.705363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3692.104904] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3692.143748] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.23/24@tcp [ 3692.201403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3697.166699] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3705.014899] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3711.063461] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3717.228498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3717.587779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3722.689916] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3723.791149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:201 to 0x240000400:225) [ 3729.984875] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3744.224150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3744.229899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3744.244082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3747.296342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3747.300609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.407671] Lustre: server umount lustre-OST0000 complete [ 3757.539107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3757.551373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3757.558540] Lustre: Skipped 2 previous similar messages [ 3759.812781] Lustre: server umount lustre-MDT0000 complete [ 3762.664590] LustreError: 127176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3762.688170] LustreError: 127176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3763.274987] LustreError: 127154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466722 with bad export cookie 3524033540914888866 [ 3763.278289] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3763.285325] LustreError: 127154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3763.560427] Lustre: server umount lustre-MDT0001 complete [ 3773.789445] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3776.411961] Key type lgssc unregistered [ 3776.659475] LNet: 130761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3776.664269] LNetError: 130761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3776.681816] LNet: Removed LNI 192.168.203.123@tcp [ 3777.531739] Key type .llcrypt unregistered [ 3777.534528] Key type ._llcrypt unregistered [ 3800.618816] Key type ._llcrypt registered [ 3800.623330] Key type .llcrypt registered [ 3800.748491] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3814.273887] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3815.288475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3815.337104] alg: No test for adler32 (adler32-zlib) [ 3816.480844] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 3816.745806] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3818.383486] Key type lgssc registered [ 3819.480622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3852.045377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3852.093112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.266340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3853.300348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3853.381563] Lustre: lustre-MDT0000: new disk, initializing [ 3853.508675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3853.530096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3857.670410] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3866.708100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.813504] Lustre: 134158:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3866.865041] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3866.870334] Lustre: Skipped 1 previous similar message [ 3866.931399] Lustre: lustre-MDT0001: new disk, initializing [ 3867.034149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3867.070330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3867.078151] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3871.068712] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3879.435450] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3884.211802] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3889.959742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3890.249541] Lustre: lustre-OST0000: new disk, initializing [ 3890.253908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3890.258974] Lustre: 136123:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3890.351662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3895.022880] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3899.982953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3899.999933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3900.157278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3902.859879] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3906.800673] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3906.993192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3910.979600] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3911.173917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3915.745071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3915.753288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3915.769151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3920.868912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3920.877299] Lustre: Skipped 1 previous similar message [ 3925.994696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3925.999912] Lustre: Skipped 1 previous similar message [ 3927.007164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3927.164382] Lustre: server umount lustre-OST0000 complete [ 3931.104536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3931.113832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3931.123531] Lustre: Skipped 2 previous similar messages [ 3931.135375] Lustre: Skipped 2 previous similar messages [ 3936.224594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3936.232431] Lustre: Skipped 1 previous similar message [ 3936.568193] Lustre: server umount lustre-MDT0000 complete [ 3939.905263] LustreError: 136121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466899 with bad export cookie 6451275140723244276 [ 3939.916434] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3940.276414] Lustre: server umount lustre-MDT0001 complete [ 3948.308179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3948.852730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3952.905231] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3954.148749] LustreError: 138343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3954.198298] LustreError: 138343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3958.260651] LustreError: 138342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3960.868568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3961.140452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3964.917809] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3971.194141] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3975.879836] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3981.999878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3982.322948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3988.782340] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3995.813507] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4000.427842] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4000.704262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4004.745740] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4005.034728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4012.937811] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4013.146299] Lustre: lustre-OST0001: new disk, initializing [ 4013.150462] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4013.157921] Lustre: 141717:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4013.276909] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4019.880082] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4021.851058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4021.861088] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4021.923563] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4027.283715] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4027.467438] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4031.751319] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4031.966711] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4037.125901] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c34d0b58000.ost_server_uuid 50 [ 4038.405607] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8c34d0b58000.ost_server_uuid in IDLE state after 0 sec [ 4047.844112] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4047.858394] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4047.862356] LustreError: Skipped 1 previous similar message [ 4047.867295] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4047.884716] Lustre: Skipped 1 previous similar message [ 4051.199921] Lustre: server umount lustre-OST0001 complete [ 4052.962877] LustreError: 142003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4052.975424] LustreError: 142003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4055.008889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4055.014878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4055.026535] Lustre: Skipped 1 previous similar message [ 4058.085352] LustreError: 142003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4058.101935] LustreError: 142003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4063.204219] LustreError: 140168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4063.219136] LustreError: 140168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4068.319107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4068.322468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4068.338548] Lustre: Skipped 6 previous similar messages [ 4068.447914] Lustre: server umount lustre-OST0000 complete [ 4073.444826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4073.457988] Lustre: Skipped 2 previous similar messages [ 4077.517495] Lustre: server umount lustre-MDT0000 complete [ 4078.561566] LustreError: 138343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4078.575455] LustreError: 138343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4080.600213] LustreError: 138324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788467039 with bad export cookie 6451275140723245158 [ 4080.603490] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4080.611035] LustreError: 138324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4080.967049] Lustre: server umount lustre-MDT0001 complete [ 4096.728893] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 4098.851267] Key type lgssc unregistered [ 4099.115505] LNet: 144718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4099.120764] LNetError: 144718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4099.137065] LNet: Removed LNI 192.168.203.123@tcp [ 4099.797789] Key type .llcrypt unregistered [ 4099.802679] Key type ._llcrypt unregistered [ 4135.062956] Key type ._llcrypt registered [ 4135.072332] Key type .llcrypt registered [ 4135.175943] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4148.610564] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4149.554739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4149.594546] alg: No test for adler32 (adler32-zlib) [ 4150.546385] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 4150.769322] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 4152.447594] Key type lgssc registered [ 4153.402160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4177.368180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4177.379628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4178.570655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4178.591593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4178.653870] Lustre: lustre-MDT0000: new disk, initializing [ 4178.711762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4178.723431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4182.154427] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4190.107361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4190.188738] Lustre: 148871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4190.237186] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4190.240752] Lustre: Skipped 1 previous similar message [ 4190.297854] Lustre: lustre-MDT0001: new disk, initializing [ 4190.364742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4190.399547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4190.405017] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4193.362454] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4200.590356] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4205.070511] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4210.150656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4210.358413] Lustre: lustre-OST0000: new disk, initializing [ 4210.361842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4210.367815] Lustre: 150839:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4210.448353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4215.197750] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4216.865563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4216.872557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4216.981340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4222.500739] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4226.965518] Lustre: Failing over lustre-OST0000 [ 4227.043721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4227.044415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4227.059706] Lustre: Skipped 1 previous similar message [ 4227.110352] Lustre: server umount lustre-OST0000 complete [ 4243.424664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4243.713871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4243.728438] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4245.911321] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4247.808024] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4247.813384] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4248.128275] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4250.877390] Lustre: Failing over lustre-MDT0000 [ 4251.000254] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 4251.130913] Lustre: server umount lustre-MDT0000 complete [ 4252.128827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4252.132450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.142766] LustreError: 150084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4252.155758] LustreError: 150084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4253.155113] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4253.156230] LustreError: 148883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4253.167054] Lustre: Skipped 1 previous similar message [ 4253.184219] LustreError: 148883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4256.096499] LustreError: 148877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4258.271954] LustreError: 148882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4258.292164] LustreError: 148882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4263.392987] LustreError: 150084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4263.409159] LustreError: 150084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4267.624412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4267.706581] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4267.883539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4267.915500] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4269.404776] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4271.269278] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4273.127460] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4273.136174] Lustre: Skipped 1 previous similar message [ 4273.151225] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4273.187491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4278.241932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4278.254983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4278.265783] Lustre: Skipped 1 previous similar message [ 4282.876900] Lustre: server umount lustre-OST0000 complete [ 4288.482899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4288.483171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4288.507575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4288.513495] Lustre: Skipped 2 previous similar messages [ 4291.402856] Lustre: server umount lustre-MDT0000 complete [ 4293.602909] LustreError: 148883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4293.619669] LustreError: 148883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4294.396300] LustreError: 148864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788467253 with bad export cookie 9019754663899504160 [ 4294.411115] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4294.412293] LustreError: 148864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4294.699087] Lustre: server umount lustre-MDT0001 complete [ 4303.734240] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 4306.120230] Key type lgssc unregistered [ 4306.351698] LNet: 154486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4306.360712] LNetError: 154486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4307.435563] LNet: Removed LNI 192.168.203.123@tcp [ 4308.294918] Key type .llcrypt unregistered [ 4308.300450] Key type ._llcrypt unregistered [ 4342.450588] Key type ._llcrypt registered [ 4342.454280] Key type .llcrypt registered [ 4342.573969] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4343.254626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4343.390272] alg: No test for adler32 (adler32-zlib) [ 4344.379514] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 4344.550126] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 4346.191511] Key type lgssc registered [ 4347.195578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4353.876409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4353.907722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4355.368926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4359.345746] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4365.797654] LustreError: 156005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4365.822819] LustreError: 156005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4367.498774] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4370.715043] LustreError: 156033:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4370.723384] LustreError: 156033:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4370.729308] LustreError: 156033:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 4370.999104] Lustre: server umount lustre-MDT0000 complete [ 4383.654199] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4391.766609] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4421.364660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.548472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4421.572836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4421.634907] Lustre: lustre-MDT0000: new disk, initializing [ 4421.685537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.717773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4425.432661] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4435.031289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4435.163756] Lustre: 159332:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4435.215414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4435.221031] Lustre: Skipped 1 previous similar message [ 4435.287196] Lustre: lustre-MDT0001: new disk, initializing [ 4435.376206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4435.419336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4435.430295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4439.439126] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4447.054847] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4452.162407] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4458.163456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4458.494639] Lustre: lustre-OST0000: new disk, initializing [ 4458.498337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4458.504136] Lustre: 161302:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4458.579549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4464.054415] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4464.197644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4464.205612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4464.361843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4471.713465] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4475.979751] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4477.291630] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 16:30:35 (1788467435) [ 4684.768903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.771246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4684.784656] Lustre: Skipped 2 previous similar messages [ 4684.792920] Lustre: Skipped 1 previous similar message [ 4688.203030] Lustre: server umount lustre-MDT0000 complete [ 4689.895682] LustreError: 159345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4689.920459] LustreError: 159345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4691.849153] LustreError: 159327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788467651 with bad export cookie 1704314985298499108 [ 4691.855735] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4691.861770] LustreError: 159327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4695.020066] LustreError: 159341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4695.025771] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.058631] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4695.068192] Lustre: Skipped 1 previous similar message [ 4701.152187] LustreError: 159339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4701.153552] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4701.179457] LustreError: 159339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4701.196057] Lustre: Skipped 1 previous similar message [ 4706.271137] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4706.611100] Lustre: server umount lustre-MDT0001 complete [ 4710.260647] LustreError: 163591:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x4:0x0].0x0 (00000000a9375109) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4716.559444] Lustre: server umount lustre-OST0000 complete [ 4724.852978] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4733.556638] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4763.849487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4764.174284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4764.223985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4764.281399] Lustre: lustre-MDT0000: new disk, initializing [ 4764.337705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4764.353520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4768.437932] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4778.717572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4778.791876] Lustre: 165955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4778.834045] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4778.837452] Lustre: Skipped 1 previous similar message [ 4778.942349] Lustre: lustre-MDT0001: new disk, initializing [ 4779.080506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4779.108723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4779.115799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4785.357031] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4794.704381] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4801.022891] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4808.176339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4808.513352] Lustre: lustre-OST0000: new disk, initializing [ 4808.521973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4808.531211] Lustre: 167922:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4808.662995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4815.055289] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4818.516468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4818.534199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4818.646349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4823.844202] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4827.994941] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4828.187934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4831.891720] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4832.135884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4834.279458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4834.281988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4834.310786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4834.322250] Lustre: Skipped 2 previous similar messages [ 4839.395641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4839.967677] Lustre: server umount lustre-OST0000 complete [ 4844.513838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4844.528222] Lustre: Skipped 1 previous similar message [ 4849.611616] Lustre: server umount lustre-MDT0000 complete [ 4849.636522] LustreError: 165967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4849.655514] LustreError: 165967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4853.647932] LustreError: 165949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788467812 with bad export cookie 1704314985298657938 [ 4853.653658] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4853.661586] LustreError: 165949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4854.018043] Lustre: server umount lustre-MDT0001 complete [ 4862.662547] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:37:00 (1788467820) [ 4885.042619] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4898.768652] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4936.256787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4936.468458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4936.491813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4936.558964] Lustre: lustre-MDT0000: new disk, initializing [ 4936.624939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4936.643594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4940.353277] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4950.438480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4950.587381] Lustre: 172681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4950.856063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4956.331365] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4966.745679] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4972.400790] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4979.164994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4979.479419] Lustre: lustre-OST0000: new disk, initializing [ 4979.482299] Lustre: Skipped 1 previous similar message [ 4979.486177] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4979.493110] Lustre: Skipped 2 previous similar messages [ 4979.505701] Lustre: 174657:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4986.140251] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4987.455422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4987.467657] Lustre: Skipped 1 previous similar message [ 4987.496070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4987.665902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4995.053809] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5003.233087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5003.235549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5003.253127] LustreError: Skipped 1 previous similar message [ 5003.253763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5003.253766] Lustre: Skipped 3 previous similar messages [ 5003.283631] Lustre: Skipped 2 previous similar messages [ 5008.835173] Lustre: server umount lustre-OST0000 complete [ 5012.448417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5012.454857] LustreError: Skipped 1 previous similar message [ 5012.460251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5018.744581] Lustre: server umount lustre-MDT0000 complete [ 5022.682305] LustreError: 172673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788467981 with bad export cookie 1704314985298658820 [ 5022.692737] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5022.701743] LustreError: 172673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5023.073393] Lustre: server umount lustre-MDT0001 complete [ 5032.140752] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:39:50 (1788467990) [ 5049.796233] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5060.692880] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5101.036608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5101.216266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5101.245299] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5101.296978] Lustre: lustre-MDT0000: new disk, initializing [ 5101.440065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5101.445080] Lustre: Skipped 2 previous similar messages [ 5101.455979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5107.598095] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5120.657956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5120.881118] Lustre: 178980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5121.315815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5126.109291] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5136.588519] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5142.358964] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5148.775621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5149.018803] Lustre: 180959:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5154.894414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5154.937779] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5154.976049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5162.761305] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5175.780802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5175.790338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5175.827918] Lustre: Skipped 1 previous similar message [ 5175.849775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5175.857204] Lustre: Skipped 9 previous similar messages [ 5177.159787] Lustre: server umount lustre-OST0000 complete [ 5182.954039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5182.966017] LustreError: Skipped 1 previous similar message [ 5187.564403] Lustre: server umount lustre-MDT0000 complete [ 5191.138601] LustreError: 178992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5191.153619] LustreError: 178992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5191.510973] LustreError: 178974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468150 with bad export cookie 1704314985298659905 [ 5191.513376] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5191.526890] LustreError: 178974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5191.946691] Lustre: server umount lustre-MDT0001 complete [ 5204.676733] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5208.223127] Key type lgssc unregistered [ 5208.618607] LNet: 182816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5208.629422] LNetError: 182816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5208.655371] LNet: Removed LNI 192.168.203.123@tcp [ 5209.674777] Key type .llcrypt unregistered [ 5209.676591] Key type ._llcrypt unregistered [ 5233.682477] Key type ._llcrypt registered [ 5233.684896] Key type .llcrypt registered [ 5233.793055] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5234.695408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5234.747022] alg: No test for adler32 (adler32-zlib) [ 5235.812717] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 5236.098240] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5237.864674] Key type lgssc registered [ 5239.247819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5247.668168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5247.719807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5249.277251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5254.654814] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5259.746127] LustreError: 184230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5259.779512] LustreError: 184230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5263.968596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5264.323455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5268.635861] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5275.436642] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5281.602899] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5290.410624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5290.709353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5294.860565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5297.528199] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5304.628623] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5329.891318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5329.891708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5329.899119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5329.907401] Lustre: Skipped 1 previous similar message [ 5332.799464] Lustre: server umount lustre-OST0000 complete [ 5340.134330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5340.149493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5340.167182] Lustre: Skipped 2 previous similar messages [ 5342.555634] Lustre: server umount lustre-MDT0000 complete [ 5345.251015] LustreError: 184226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5345.284184] LustreError: 184226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5346.614427] LustreError: 184208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468305 with bad export cookie 965789116817949900 [ 5346.619940] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5346.636612] LustreError: 184208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5347.159661] Lustre: server umount lustre-MDT0001 complete [ 5361.072118] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5364.655422] Key type lgssc unregistered [ 5364.956184] LNet: 187785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5364.960198] LNetError: 187785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5364.977975] LNet: Removed LNI 192.168.203.123@tcp [ 5365.832855] Key type .llcrypt unregistered [ 5365.837283] Key type ._llcrypt unregistered [ 5391.806496] Key type ._llcrypt registered [ 5391.808432] Key type .llcrypt registered [ 5391.884329] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5392.763992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5392.803145] alg: No test for adler32 (adler32-zlib) [ 5393.837254] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 5394.040699] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5395.711283] Key type lgssc registered [ 5396.945545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5406.384521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5406.406910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5408.157604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5414.190045] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5418.472904] LustreError: 189198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5418.497389] LustreError: 189198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5423.585621] LustreError: 189197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5424.507781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5424.964375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5430.212565] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5437.661151] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5443.223601] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5450.196975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5450.678243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.748585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5457.189787] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5464.515578] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5471.097294] Lustre: lustre-OST0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 5472.224289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5472.235664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5472.265492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5474.276490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5474.280616] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5474.293430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5476.205726] Lustre: server umount lustre-OST0000 complete [ 5483.479381] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8c34d02b6000.ost_server_uuid 50 [ 5484.863602] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5499.872973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5499.880404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5499.892022] Lustre: Skipped 1 previous similar message [ 5499.902174] Lustre: Skipped 1 previous similar message [ 5505.006617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5505.016658] Lustre: Skipped 1 previous similar message [ 5505.153282] Lustre: server umount lustre-MDT0000 complete [ 5509.033831] LustreError: 191776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5509.056136] LustreError: 191776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5509.143065] LustreError: 189179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468468 with bad export cookie 2335456561420634864 [ 5509.160449] LustreError: 189179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5509.165488] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5509.483839] Lustre: server umount lustre-MDT0001 complete [ 5519.096648] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 16:47:56 (1788468476) [ 5527.362425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5527.900780] LustreError: 193115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5528.013846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5533.095308] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5533.155953] LustreError: 193116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5541.513037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.751676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5547.278094] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5555.614550] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5562.800890] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5569.923877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5570.290966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5573.379640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5577.001588] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5584.322696] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5591.026745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5591.273981] Lustre: lustre-OST0001: new disk, initializing [ 5591.281505] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5591.292283] Lustre: 195958:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5591.395902] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5597.798048] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5598.383512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5598.408996] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5598.515207] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5606.851329] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5612.512386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5612.518905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.537643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5615.837107] Lustre: server umount lustre-OST0000 complete [ 5618.015231] LustreError: 194939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5618.040432] LustreError: 194939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5622.656248] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5622.936589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5627.420852] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5627.707643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5634.532506] LustreError: 194939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5634.567832] LustreError: 194939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 5644.771156] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5644.786444] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5644.793348] Lustre: Skipped 1 previous similar message [ 5644.805425] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5644.809582] Lustre: Skipped 4 previous similar messages [ 5648.273661] Lustre: server umount lustre-OST0001 complete [ 5654.496103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5654.504796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5654.523031] Lustre: Skipped 1 previous similar message [ 5658.898922] Lustre: server umount lustre-MDT0000 complete [ 5662.084301] LustreError: 193096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468621 with bad export cookie 2335456561420635935 [ 5662.088544] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5662.104312] LustreError: 193096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5662.477211] Lustre: server umount lustre-MDT0001 complete [ 5685.781812] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 16:50:43 (1788468643) [ 5693.258764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5693.593701] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5693.666308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5693.922140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5698.272549] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5706.891887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5706.962344] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5706.997277] Lustre: 199405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5712.146665] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5718.973474] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5725.587526] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5732.104214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5732.341544] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5738.691054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5741.356475] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5749.296476] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5755.495985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5755.540120] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5761.242195] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5761.415148] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5768.028570] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5771.757424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5771.770779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5771.778283] Lustre: Skipped 2 previous similar messages [ 5771.789795] Lustre: Skipped 4 previous similar messages [ 5776.748802] Lustre: server umount lustre-OST0000 complete [ 5776.870778] LustreError: 201258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.890661] LustreError: 201258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5792.233200] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5792.244310] LustreError: Skipped 1 previous similar message [ 5798.002816] Lustre: server umount lustre-OST0001 complete [ 5804.513206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5804.542042] Lustre: Skipped 2 previous similar messages [ 5808.885931] Lustre: server umount lustre-MDT0000 complete [ 5812.763171] LustreError: 200162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468771 with bad export cookie 2335456561420637342 [ 5812.768896] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5812.783782] LustreError: 200162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5813.460765] Lustre: server umount lustre-MDT0001 complete [ 5834.782376] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 16:53:13 (1788468793) [ 5848.328648] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5857.071686] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5888.396897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5888.657073] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5888.708265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5888.815642] Lustre: lustre-MDT0000: new disk, initializing [ 5888.977270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5888.983834] Lustre: Skipped 3 previous similar messages [ 5889.004986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5893.773768] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5903.539940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5903.706845] Lustre: 207500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5903.750728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5903.755085] Lustre: Skipped 1 previous similar message [ 5903.814740] Lustre: lustre-MDT0001: new disk, initializing [ 5903.907728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5903.918229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5908.157399] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5918.545424] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5925.772294] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5933.508418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5933.715716] Lustre: lustre-OST0000: new disk, initializing [ 5933.719624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5933.725635] Lustre: 209467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5939.599491] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5942.940792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5942.966822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5943.116030] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5949.429232] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5954.583029] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5954.867579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5959.344982] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5959.571456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5963.747872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5963.748185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5963.765866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5963.765877] Lustre: Skipped 8 previous similar messages [ 5963.788318] Lustre: Skipped 2 previous similar messages [ 5963.830903] LustreError: Skipped 1 previous similar message [ 5967.782185] Lustre: server umount lustre-OST0000 complete [ 5977.792073] Lustre: server umount lustre-MDT0000 complete [ 5979.108866] LustreError: 207510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5979.137409] LustreError: 207510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 5981.950653] LustreError: 207492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468941 with bad export cookie 2335456561420639120 [ 5981.952403] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5981.962628] LustreError: 207492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5982.254912] Lustre: server umount lustre-MDT0001 complete [ 5992.356277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5998.424812] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6008.680361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6015.568612] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6025.235052] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6032.083227] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6040.237373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6040.846452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6040.855217] Lustre: Skipped 4 previous similar messages [ 6047.886405] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6055.022959] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6059.653907] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6064.226234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 6068.146881] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6068.332559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6070.753151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6070.759722] LustreError: Skipped 1 previous similar message [ 6076.535657] Lustre: server umount lustre-OST0000 complete [ 6082.743312] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6087.033847] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6116.133686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6123.448308] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6123.907163] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.927949] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6132.095493] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6135.426907] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6139.304153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.348355] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6144.481249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6144.489828] LustreError: Skipped 1 previous similar message [ 6144.492532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6144.494870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6144.503569] Lustre: Skipped 5 previous similar messages [ 6144.509649] Lustre: Skipped 6 previous similar messages [ 6146.817666] Lustre: server umount lustre-OST0000 complete [ 6157.284975] Lustre: server umount lustre-MDT0000 complete [ 6159.842843] LustreError: 211681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6159.885331] LustreError: 211681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6161.318672] LustreError: 211665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469120 with bad export cookie 2335456561420640009 [ 6161.330088] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6161.334803] LustreError: 211665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6161.793871] Lustre: server umount lustre-MDT0001 complete [ 6169.994855] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:58:47 (1788469127) [ 6175.550912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.157034] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6187.924751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6193.025147] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6200.443168] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6206.574264] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6212.683156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6219.657673] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6228.630197] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6234.754042] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6234.991900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6239.498944] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6239.714499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6246.643359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6246.851893] Lustre: lustre-OST0001: new disk, initializing [ 6246.858341] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6246.866196] Lustre: 221052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6253.202952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6253.218804] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6253.364053] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6254.048746] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6263.518288] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6268.662531] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6268.927769] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6273.922677] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6274.202482] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6279.136354] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6279.153154] LustreError: Skipped 2 previous similar messages [ 6290.399132] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6290.512321] Lustre: server umount lustre-OST0001 complete [ 6296.789524] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6297.053797] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6301.296099] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6301.484457] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6329.843689] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6330.116066] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6330.128387] Lustre: Skipped 5 previous similar messages [ 6335.469247] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6335.484125] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6335.485605] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6335.514936] Lustre: Skipped 1 previous similar message [ 6337.400676] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6346.182642] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6348.412103] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6356.816497] Lustre: server umount lustre-OST0001 complete [ 6379.719587] Lustre: server umount lustre-OST0000 complete [ 6389.651346] LustreError: 224640:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000c029cf91) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6389.723193] Lustre: server umount lustre-MDT0000 complete [ 6393.442447] LustreError: 217661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469352 with bad export cookie 2335456561420640975 [ 6393.450226] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6393.457232] LustreError: 217661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6399.870271] Lustre: server umount lustre-MDT0001 complete [ 6425.021185] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:03:02 (1788469382) [ 6432.442110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6432.734653] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6432.771723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6437.702296] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6446.520523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6446.675143] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6446.697142] Lustre: 226311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6451.228163] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6459.904435] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6466.939155] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6473.557823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6473.759414] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6480.472716] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6488.174375] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6500.170496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6500.264983] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6512.089592] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6520.682914] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6522.493492] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6527.081320] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6527.328259] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6531.384048] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6531.625693] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6537.871407] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c34d861a800.ost_server_uuid 50 [ 6539.462192] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8c34d861a800.ost_server_uuid in IDLE state after 0 sec [ 6540.602415] Lustre: Permanently deactivating lustre-OST0001 [ 6540.619316] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6543.916991] Lustre: Permanently reactivating lustre-OST0001 [ 6549.984341] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6549.987372] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6549.999541] LustreError: Skipped 3 previous similar messages [ 6550.003744] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6550.029026] Lustre: Skipped 11 previous similar messages [ 6550.059856] Lustre: Skipped 20 previous similar messages [ 6554.549370] Lustre: server umount lustre-OST0001 complete [ 6555.115631] LustreError: 228161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6555.139189] LustreError: 228161:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 34 previous similar messages [ 6564.888969] Lustre: server umount lustre-OST0000 complete [ 6574.539208] Lustre: server umount lustre-MDT0000 complete [ 6577.825055] LustreError: 226304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469536 with bad export cookie 2335456561420642263 [ 6577.832226] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6577.837109] LustreError: 226304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6578.344534] Lustre: server umount lustre-MDT0001 complete [ 6601.191540] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:05:59 (1788469559) [ 6607.820681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6608.148511] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6608.187404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6608.198788] Lustre: Skipped 5 previous similar messages [ 6612.520572] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6622.139618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6622.192763] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6622.217374] Lustre: 233004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6627.819130] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6635.292160] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6640.071364] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6646.942030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6647.240450] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6647.262509] Lustre: Permanently deactivating lustre-OST0000 [ 6647.274975] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6652.992918] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6653.106176] LustreError: 235171:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 6654.152355] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6661.295469] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6668.263354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6668.321779] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6675.143592] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6677.641128] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6682.951227] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6684.588758] Lustre: Permanently reactivating lustre-OST0000 [ 6684.606561] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6684.610977] Lustre: Skipped 2 previous similar messages [ 6684.720751] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6684.743079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6684.761902] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6684.790625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6694.432726] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6694.442995] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6694.453958] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6705.733633] Lustre: server umount lustre-OST0001 complete [ 6724.070806] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6724.354589] Lustre: server umount lustre-OST0000 complete [ 6734.700979] Lustre: server umount lustre-MDT0000 complete [ 6739.358583] LustreError: 232997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469698 with bad export cookie 2335456561420643915 [ 6739.360850] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6739.374026] LustreError: 232997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6740.009950] Lustre: server umount lustre-MDT0001 complete [ 6752.921794] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6756.266294] Key type lgssc unregistered [ 6756.724456] LNet: 237893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6756.733212] LNetError: 237893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6756.765994] LNet: Removed LNI 192.168.203.123@tcp [ 6758.251320] Key type .llcrypt unregistered [ 6758.253365] Key type ._llcrypt unregistered [ 6785.837918] Key type ._llcrypt registered [ 6785.839863] Key type .llcrypt registered [ 6785.932532] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6787.436570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6787.488043] alg: No test for adler32 (adler32-zlib) [ 6788.586615] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 6788.853530] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6790.559317] Key type lgssc registered [ 6791.794302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6801.270397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6801.280988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6802.722239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6807.590883] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6812.128882] LustreError: 239355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6812.143646] LustreError: 239355:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6815.849846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6815.979921] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6816.005842] Lustre: 239343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6816.015928] Lustre: Permanently deactivating lustre-MDT0001 [ 6816.030333] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6816.426851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6822.147168] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6824.497766] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6831.547759] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6837.405797] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6844.152312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6844.581938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6847.652730] LustreError: 241200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6847.699908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6851.363359] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6853.105425] LustreError: 241371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6858.219529] LustreError: 241201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6859.657745] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6866.411280] LustreError: 241202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6866.431284] LustreError: 241202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6866.942353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6867.260420] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6872.559777] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6872.584587] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6872.849830] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6880.282059] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6884.250731] Lustre: Permanently reactivating lustre-MDT0001 [ 6884.271588] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6884.278696] Lustre: Skipped 2 previous similar messages [ 6886.870208] Lustre: lustre-MDT0001: Client f01d604d-c865-4360-b13f-e985ed107d8d (at 192.168.203.23@tcp) reconnecting [ 6892.221613] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6892.241651] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6892.246965] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6892.264418] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6893.526511] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6895.243230] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6901.579941] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6901.816030] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6906.552081] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6906.824086] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6909.731570] Lustre: Permanently deactivating lustre-MDT0001 [ 6909.749577] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6909.757786] Lustre: Skipped 2 previous similar messages [ 6919.780331] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6933.998900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6933.999962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6934.026991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6934.049643] Lustre: Skipped 2 previous similar messages [ 6936.446703] Lustre: server umount lustre-MDT0000 complete [ 6939.107114] LustreError: 239352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6939.117055] LustreError: 239352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6943.849976] LustreError: 240094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469903 with bad export cookie 5081473097435232380 [ 6943.859713] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6943.866150] LustreError: 240094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6944.537459] Lustre: server umount lustre-MDT0001 complete [ 6952.019384] Lustre: server umount lustre-OST0000 complete [ 6965.838396] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:12:03 (1788469923) [ 6976.249231] Lustre: server umount lustre-OST0001 complete [ 6984.359069] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 6999.310936] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7042.731689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7043.028463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7043.036679] Lustre: Skipped 2 previous similar messages [ 7043.060476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7043.135052] Lustre: lustre-MDT0000: new disk, initializing [ 7043.261768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7043.291890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7048.688584] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7061.588100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7061.658468] Lustre: 247640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7061.703495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7061.708670] Lustre: Skipped 1 previous similar message [ 7061.757457] Lustre: lustre-MDT0001: new disk, initializing [ 7061.827135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7061.892081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7061.912865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7067.513765] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7077.782262] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7083.598368] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7090.894317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7091.236044] Lustre: lustre-OST0000: new disk, initializing [ 7091.244642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7091.251275] Lustre: 249609:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7091.383283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7097.956966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7097.969110] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7098.078857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7098.125873] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7107.836627] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7110.104571] LustreError: 250328:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7117.355115] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7117.460496] Lustre: lustre-OST0001: new disk, initializing [ 7117.464371] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7117.469892] Lustre: 250723:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7120.175178] LustreError: 250328:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7120.221927] LustreError: 247646:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7123.272277] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7127.158361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7127.169336] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7127.347940] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7130.256733] LustreError: 247646:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7134.067935] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7135.783972] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7142.884107] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7142.895706] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7142.903840] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7142.914036] Lustre: Skipped 2 previous similar messages [ 7142.936601] Lustre: Skipped 4 previous similar messages [ 7144.336419] Lustre: server umount lustre-OST0001 complete [ 7148.008453] LustreError: 249594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7148.039950] LustreError: 249594:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7162.338515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7162.352796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7162.367964] Lustre: Skipped 1 previous similar message [ 7162.397992] Lustre: Skipped 1 previous similar message [ 7167.465024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7167.478425] Lustre: Skipped 1 previous similar message [ 7168.046607] Lustre: server umount lustre-OST0000 complete [ 7172.604819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7172.637530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7174.625866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7174.640288] LustreError: Skipped 1 previous similar message [ 7179.032670] Lustre: server umount lustre-MDT0000 complete [ 7182.816855] LustreError: 247650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7182.845073] LustreError: 247650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7183.494494] LustreError: 248435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470142 with bad export cookie 5081473097435234977 [ 7183.496417] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7183.507079] LustreError: 248435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7189.764517] Lustre: server umount lustre-MDT0001 complete [ 7198.373262] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7200.990071] Key type lgssc unregistered [ 7201.254449] LNet: 252755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7201.269542] LNetError: 252755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7202.341955] LNet: Removed LNI 192.168.203.123@tcp [ 7203.392144] Key type .llcrypt unregistered [ 7203.397242] Key type ._llcrypt unregistered [ 7246.909792] Key type ._llcrypt registered [ 7246.913155] Key type .llcrypt registered [ 7247.059449] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7249.419120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7249.463776] alg: No test for adler32 (adler32-zlib) [ 7250.580544] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 7250.906884] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7252.647263] Key type lgssc registered [ 7253.664879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7263.034534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7263.048716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7264.301966] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7264.331609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7264.460067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7268.560917] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7276.670753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7276.801859] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7276.838986] Lustre: 254959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7277.123428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7282.573862] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7290.189828] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7296.994933] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7303.557049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7303.705712] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7303.879236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7309.949471] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7314.054532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7317.088796] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7370.209841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7370.219661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7370.243340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7370.732545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7370.748263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7370.786547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7376.031247] Lustre: server umount lustre-OST0000 complete [ 7382.871783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7400.592609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7401.642069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7410.873241] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7410.912490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7410.919390] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7410.950273] Lustre: Skipped 1 previous similar message [ 7413.725642] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7427.581144] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7472.611489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7472.614274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7472.614645] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7472.623672] Lustre: Skipped 1 previous similar message [ 7474.774481] Lustre: server umount lustre-OST0000 complete [ 7481.824869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7481.834025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7481.867430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7481.882514] Lustre: Skipped 1 previous similar message [ 7484.836423] Lustre: server umount lustre-MDT0000 complete [ 7488.883473] LustreError: 256822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470448 with bad export cookie 6558031018492427740 [ 7488.891486] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7488.895834] LustreError: 256822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7489.586091] Lustre: server umount lustre-MDT0001 complete [ 7503.424762] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7506.927539] Key type lgssc unregistered [ 7507.459662] LNet: 260385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7507.469845] LNetError: 260385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7507.494326] LNet: Removed LNI 192.168.203.123@tcp [ 7508.639588] Key type .llcrypt unregistered [ 7508.641097] Key type ._llcrypt unregistered [ 7533.517865] Key type ._llcrypt registered [ 7533.521362] Key type .llcrypt registered [ 7533.608540] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7534.587826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7534.680289] alg: No test for adler32 (adler32-zlib) [ 7535.861930] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 7536.181727] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7537.847157] Key type lgssc registered [ 7539.123242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7546.924447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7546.961116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7548.602529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7553.581709] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7554.023332] LustreError: 261801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7554.055351] LustreError: 261801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7558.114038] LustreError: 261800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7562.016582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7562.416462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7566.637374] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7574.436714] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7579.953220] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7586.854784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7587.175546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7592.860627] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7593.337669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7599.584786] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7624.164588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7624.173050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7624.186513] LustreError: Skipped 1 previous similar message [ 7624.188397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7624.230712] Lustre: Skipped 1 previous similar message [ 7628.818721] Lustre: server umount lustre-OST0000 complete [ 7634.400295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7634.401810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7634.414075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7634.429896] Lustre: Skipped 1 previous similar message [ 7634.442486] Lustre: Skipped 2 previous similar messages [ 7638.845192] Lustre: server umount lustre-MDT0000 complete [ 7639.520263] LustreError: 261797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7639.536491] LustreError: 261797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7642.459324] LustreError: 265155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470601 with bad export cookie 5204613489898982358 [ 7642.465063] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7642.473157] LustreError: 265155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7642.822733] Lustre: server umount lustre-MDT0001 complete [ 7653.414609] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7656.359607] Key type lgssc unregistered [ 7656.662811] LNet: 266107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7656.680238] LNetError: 266107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7656.700406] LNet: Removed LNI 192.168.203.123@tcp [ 7657.573237] Key type .llcrypt unregistered [ 7657.578565] Key type ._llcrypt unregistered [ 7675.717387] Key type ._llcrypt registered [ 7675.719888] Key type .llcrypt registered [ 7675.820131] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7677.711245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7677.776892] alg: No test for adler32 (adler32-zlib) [ 7678.886207] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 7679.171993] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7680.935155] Key type lgssc registered [ 7682.784775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7692.005379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7692.042074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7693.727099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7699.124201] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7703.008564] LustreError: 267327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7703.031178] LustreError: 267327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7708.128465] LustreError: 267326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7708.693915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7709.186385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7714.785669] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7723.149174] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7729.356921] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7737.163801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7737.490033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7738.567937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7744.343414] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7752.516599] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7767.024799] Lustre: server umount lustre-OST0000 complete [ 7774.690029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7774.692746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7774.724114] Lustre: Skipped 1 previous similar message [ 7774.738378] Lustre: Skipped 1 previous similar message [ 7779.816406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7779.831237] Lustre: Skipped 1 previous similar message [ 7781.907601] Lustre: server umount lustre-MDT0000 complete [ 7784.927671] Lustre: 266638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788470728/real 1788470728] req@00000000b085484b x1875347387741184/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788470744 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7784.931518] LustreError: 267323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7784.960962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7785.012274] LustreError: 267323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7785.985791] LustreError: 267308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470745 with bad export cookie 17619908243873214896 [ 7785.997236] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7785.999433] LustreError: 267308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7786.380194] Lustre: server umount lustre-MDT0001 complete [ 7797.684824] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7800.700335] Key type lgssc unregistered [ 7801.064080] LNet: 271002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7801.079978] LNetError: 271002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7801.103996] LNet: Removed LNI 192.168.203.123@tcp [ 7802.177453] Key type .llcrypt unregistered [ 7802.181668] Key type ._llcrypt unregistered [ 7828.031478] Key type ._llcrypt registered [ 7828.033182] Key type .llcrypt registered [ 7828.152581] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7829.022532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7829.070115] alg: No test for adler32 (adler32-zlib) [ 7830.315629] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 7830.597787] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7832.311240] Key type lgssc registered [ 7833.445710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7841.235723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7841.250441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7842.797836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7847.309870] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7852.006721] LustreError: 272416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7852.025454] LustreError: 272416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7855.698327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7856.084089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7861.492569] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7870.279724] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7875.878038] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7883.040727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7883.604651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7887.679567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7890.065759] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7898.687408] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7923.680870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7923.694991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7923.719747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7926.763266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7926.770321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7926.776812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7929.659845] Lustre: server umount lustre-OST0000 complete [ 7936.993540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7937.000480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7937.010657] Lustre: Skipped 1 previous similar message [ 7937.027130] Lustre: Skipped 2 previous similar messages [ 7939.358145] Lustre: server umount lustre-MDT0000 complete [ 7942.113275] LustreError: 273143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7942.148411] LustreError: 273143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7942.831911] LustreError: 272396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470901 with bad export cookie 2737052252773849154 [ 7942.837279] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7942.841428] LustreError: 272396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7943.173312] Lustre: server umount lustre-MDT0001 complete [ 7953.215727] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7955.948467] Key type lgssc unregistered [ 7956.232591] LNet: 276724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7956.242348] LNetError: 276724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7956.256591] LNet: Removed LNI 192.168.203.123@tcp [ 7957.052667] Key type .llcrypt unregistered [ 7957.054751] Key type ._llcrypt unregistered [ 7973.021415] Key type ._llcrypt registered [ 7973.027651] Key type .llcrypt registered [ 7973.130049] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7974.022762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7974.105918] alg: No test for adler32 (adler32-zlib) [ 7975.172596] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 7975.549036] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7977.247173] Key type lgssc registered [ 7978.506699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7986.942706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7986.990354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.769258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7993.787724] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7997.934278] LustreError: 277959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7997.948139] LustreError: 277959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8002.575903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8002.976165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8007.575636] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8015.191315] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8020.748701] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8027.342274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8027.720840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8033.830072] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8033.867698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8041.427659] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8054.246614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8054.253349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8054.255147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8054.267128] Lustre: Skipped 1 previous similar message [ 8058.172713] Lustre: server umount lustre-OST0000 complete [ 8064.483925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8064.488443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8064.498670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8064.509039] Lustre: Skipped 1 previous similar message [ 8064.528946] Lustre: Skipped 2 previous similar messages [ 8067.862867] Lustre: server umount lustre-MDT0000 complete [ 8069.606215] LustreError: 277949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8069.630544] LustreError: 277949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8070.964890] LustreError: 277924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471030 with bad export cookie 5865640947141707309 [ 8070.978552] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8070.985686] LustreError: 277924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8071.401166] Lustre: server umount lustre-MDT0001 complete [ 8083.076336] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8085.835655] Key type lgssc unregistered [ 8086.104374] LNet: 281660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8086.109737] LNetError: 281660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8086.126678] LNet: Removed LNI 192.168.203.123@tcp [ 8087.000496] Key type .llcrypt unregistered [ 8087.003226] Key type ._llcrypt unregistered [ 8101.899401] Key type ._llcrypt registered [ 8101.902493] Key type .llcrypt registered [ 8101.995663] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8120.863819] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8135.449681] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8136.326265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8136.360595] alg: No test for adler32 (adler32-zlib) [ 8137.354872] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 8137.516526] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8139.175669] Key type lgssc registered [ 8140.347810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8175.490943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8175.547203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8176.814822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8176.843788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8176.945759] Lustre: lustre-MDT0000: new disk, initializing [ 8177.099709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8177.121875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8181.527481] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8192.045292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8192.223347] Lustre: 285211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8192.286080] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8192.290306] Lustre: Skipped 1 previous similar message [ 8192.357756] Lustre: lustre-MDT0001: new disk, initializing [ 8192.454872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8192.503641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8192.510845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8196.989726] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8205.953065] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8211.713751] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8218.662829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8219.026065] Lustre: lustre-OST0000: new disk, initializing [ 8219.035630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8219.047345] Lustre: 287178:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8219.154417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8225.551533] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8228.921388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8228.935037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8229.072686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8235.191863] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8240.764296] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8241.057572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8246.161662] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8246.459754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8249.831392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8249.837357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8249.840588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8249.845466] Lustre: Skipped 1 previous similar message [ 8255.046838] Lustre: server umount lustre-OST0000 complete [ 8259.039824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8259.047311] LustreError: Skipped 1 previous similar message [ 8259.056249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8259.069654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8259.080170] Lustre: Skipped 1 previous similar message [ 8260.073591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8260.109287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8260.114902] Lustre: Skipped 1 previous similar message [ 8265.188299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8265.191235] Lustre: Skipped 1 previous similar message [ 8270.305347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8270.313417] Lustre: Skipped 1 previous similar message [ 8273.375247] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8273.570575] Lustre: server umount lustre-MDT0000 complete [ 8275.431501] LustreError: 286102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8275.464718] LustreError: 286102:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8277.548946] LustreError: 285203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471236 with bad export cookie 8950468854966050547 [ 8277.561116] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8277.931959] Lustre: server umount lustre-MDT0001 complete [ 8289.319444] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:34:06 (1788471246) [ 8296.529424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8297.082636] LustreError: 289593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8297.176249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8302.493318] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8302.569746] LustreError: 289594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8307.689161] LustreError: 289593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8312.269440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8312.787413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8317.931038] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8325.119361] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8330.828329] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8337.669085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8338.066162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8344.307597] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8351.598527] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8368.103823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8368.110206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8368.122835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8368.135728] LustreError: Skipped 1 previous similar message [ 8368.149411] Lustre: Skipped 1 previous similar message [ 8372.656420] Lustre: server umount lustre-OST0000 complete [ 8378.338156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8378.355493] Lustre: Skipped 2 previous similar messages [ 8388.578063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8388.584225] Lustre: Skipped 5 previous similar messages [ 8391.135132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8391.253275] Lustre: server umount lustre-MDT0000 complete [ 8393.703749] LustreError: 290578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8393.723663] LustreError: 290578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8394.477843] LustreError: 290328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471353 with bad export cookie 8950468854966051429 [ 8394.479647] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8394.503808] LustreError: 290328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8394.764296] Lustre: server umount lustre-MDT0001 complete [ 8406.604387] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8411.032570] Key type lgssc unregistered [ 8411.426719] LNet: 293150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8411.430380] LNetError: 293150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8411.472107] LNet: Removed LNI 192.168.203.123@tcp [ 8412.660521] Key type .llcrypt unregistered [ 8412.663419] Key type ._llcrypt unregistered [ 8445.700725] Key type ._llcrypt registered [ 8445.704766] Key type .llcrypt registered [ 8445.813557] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8463.856661] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8465.369872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8465.409286] alg: No test for adler32 (adler32-zlib) [ 8466.548719] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 8466.880083] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8468.672044] Key type lgssc registered [ 8469.980769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8518.613463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8518.670348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8520.027483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8520.085322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8520.136234] Lustre: lustre-MDT0000: new disk, initializing [ 8520.216155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8520.235330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8524.282651] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8534.004116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8534.158363] Lustre: 296977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8534.255686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8534.266637] Lustre: Skipped 1 previous similar message [ 8534.346738] Lustre: lustre-MDT0001: new disk, initializing [ 8534.468287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8534.546037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8534.575531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8539.028515] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8548.867033] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8554.687605] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8561.779758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8562.068364] Lustre: lustre-OST2710: new disk, initializing [ 8562.073397] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8562.081145] Lustre: 298943:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8562.141563] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8568.102236] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8568.371417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8568.389799] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8568.471961] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8575.769811] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8581.501163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8581.655248] Lustre: lustre-OST03e8: new disk, initializing [ 8581.658479] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8581.663198] Lustre: 300008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8581.739197] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8586.078702] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8588.406553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8588.422257] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8588.511678] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8592.356690] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8599.289927] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8599.548197] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8603.543262] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8603.744564] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8607.852412] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8608.085849] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8611.919782] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8612.146967] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8619.493809] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8619.508496] Lustre: Skipped 1 previous similar message [ 8619.578209] Lustre: server umount lustre-MDT0000 complete [ 8622.990048] LustreError: 300007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471582 with bad export cookie 15090412418235560089 [ 8622.994783] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8623.001855] LustreError: 300007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8623.411093] Lustre: server umount lustre-MDT0001 complete [ 8637.380704] Lustre: server umount lustre-OST2710 complete [ 8640.481626] Lustre: 294774:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788471583/real 1788471583] req@00000000040d9959 x1875348214054144/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788471599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8640.524944] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8640.543903] Lustre: Skipped 2 previous similar messages [ 8640.989536] Lustre: server umount lustre-OST03e8 complete [ 8657.191229] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8666.507643] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8700.246367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8700.552513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8700.604349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8700.718939] Lustre: lustre-MDT0000: new disk, initializing [ 8700.791411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8700.814409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8705.170643] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8716.933727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8717.113428] Lustre: 305261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8717.170696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8717.173642] Lustre: Skipped 1 previous similar message [ 8717.253559] Lustre: lustre-MDT0001: new disk, initializing [ 8717.348923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8717.441748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8717.461174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8721.878483] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8731.219446] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8736.829416] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8743.798967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8744.086234] Lustre: lustre-OST0000: new disk, initializing [ 8744.091860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8744.098276] Lustre: 307228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8744.173989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8750.949278] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8754.223821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8754.234081] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8754.339952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8759.865860] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8764.947352] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8765.196874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8769.935934] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8770.248477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8775.144150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8775.153219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8775.172315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8775.179413] Lustre: Skipped 1 previous similar message [ 8780.257113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8780.265793] Lustre: Skipped 1 previous similar message [ 8785.388183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8785.399083] Lustre: Skipped 1 previous similar message [ 8786.400483] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8786.676731] Lustre: server umount lustre-OST0000 complete [ 8794.081255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8794.091580] LustreError: Skipped 1 previous similar message [ 8794.095944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.114128] Lustre: Skipped 1 previous similar message [ 8794.121772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8796.994751] Lustre: server umount lustre-MDT0000 complete [ 8800.246577] LustreError: 305252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471759 with bad export cookie 15090412418235561776 [ 8800.251117] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8800.257863] LustreError: 305252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8800.562178] Lustre: server umount lustre-MDT0001 complete [ 8809.003737] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:42:46 (1788471766) [ 8810.668329] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8812.340904] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:42:50 (1788471770) [ 8816.081561] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8842.293849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8842.546517] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8842.564262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8842.699158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8847.346222] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8855.289462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8855.399587] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8855.439423] Lustre: 311087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8859.197977] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8879.862287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8880.042975] Lustre: Denying initial registration attempt from nid 192.168.203.123@tcp, specified as failover [ 8880.058845] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8880.073714] LustreError: 312539:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 8880.089426] LustreError: 312539:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8880.106131] LustreError: 312539:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8880.185127] Lustre: server umount lustre-OST0000 complete [ 8880.194174] LustreError: 312539:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8886.752024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8886.758826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8886.769035] Lustre: Skipped 1 previous similar message [ 8886.775897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8886.781784] Lustre: Skipped 2 previous similar messages [ 8892.169347] Lustre: server umount lustre-MDT0000 complete [ 8893.923403] LustreError: 311092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8893.956764] LustreError: 311092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8895.151233] LustreError: 311077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471854 with bad export cookie 15090412418235562658 [ 8895.160040] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8895.165594] LustreError: 311077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8895.441403] Lustre: server umount lustre-MDT0001 complete [ 8906.668761] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8909.923886] Key type lgssc unregistered [ 8910.236614] LNet: 313658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8910.247885] LNetError: 313658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8910.262152] LNet: Removed LNI 192.168.203.123@tcp [ 8911.100331] Key type .llcrypt unregistered [ 8911.105949] Key type ._llcrypt unregistered [ 8923.420585] Key type ._llcrypt registered [ 8923.424771] Key type .llcrypt registered [ 8923.503212] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8924.567312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8924.593355] alg: No test for adler32 (adler32-zlib) [ 8925.828599] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 8926.038551] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8927.722360] Key type lgssc registered [ 8928.850102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8974.893788] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8982.097246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8982.137728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8983.392636] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8983.407271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8983.570017] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8987.782533] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8995.974453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8996.050207] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8996.107123] Lustre: 316420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8996.425777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9000.391848] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9006.410193] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9010.928951] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9029.292852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9029.466824] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9029.649155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9034.877095] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9040.831423] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9043.427652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9043.449163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9043.451869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9043.459843] Lustre: Skipped 1 previous similar message [ 9044.808039] Lustre: server umount lustre-OST0000 complete [ 9048.547083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9048.548126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9048.556764] Lustre: Skipped 2 previous similar messages [ 9053.665441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9053.684856] Lustre: Skipped 2 previous similar messages [ 9058.784341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9058.790426] Lustre: Skipped 1 previous similar message [ 9062.367172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9062.482652] Lustre: server umount lustre-MDT0000 complete [ 9065.794250] LustreError: 316410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472024 with bad export cookie 4155727258494623421 [ 9065.799154] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9065.809943] LustreError: 316410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9066.184756] Lustre: server umount lustre-MDT0001 complete [ 9076.021896] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 9078.539233] Key type lgssc unregistered [ 9078.759921] LNet: 319992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9078.768851] LNetError: 319992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9078.788645] LNet: Removed LNI 192.168.203.123@tcp [ 9079.642972] Key type .llcrypt unregistered [ 9079.646779] Key type ._llcrypt unregistered [ 9099.973634] Key type ._llcrypt registered [ 9099.977563] Key type .llcrypt registered [ 9100.057262] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9100.819058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9100.917274] alg: No test for adler32 (adler32-zlib) [ 9101.957377] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 9102.145653] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 9103.817766] Key type lgssc registered [ 9104.791042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9111.120596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9111.138589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.549650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9116.291450] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9122.785707] LustreError: 321384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9122.800634] LustreError: 321384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9123.247874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9123.601224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9127.099391] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9133.820828] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9138.423741] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9145.713838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9146.136086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9151.932618] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9158.281918] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9166.209460] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 9167.849940] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9167.854350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9167.863149] Lustre: Skipped 1 previous similar message [ 9167.876204] Lustre: Skipped 2 previous similar messages [ 9171.295746] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 9172.522495] Lustre: server umount lustre-MDT0000 complete [ 9172.972099] LustreError: 321380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9172.989555] LustreError: 321380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9175.441735] LustreError: 321364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472134 with bad export cookie 186700505358023620 [ 9175.444714] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9175.451124] LustreError: 321364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9176.411493] LustreError: 321377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9176.453302] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 9178.090603] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9178.098302] Lustre: Skipped 1 previous similar message [ 9180.135482] LustreError: 322817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9180.149299] LustreError: 322817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9181.534156] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 9181.545941] Lustre: Skipped 2 previous similar messages [ 9181.817983] Lustre: server umount lustre-MDT0001 complete [ 9187.310236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9194.766383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9200.909499] LustreError: 323215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9200.925481] LustreError: 323215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9200.976345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9201.039274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:257) [ 9204.709631] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9210.214614] LustreError: 323216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9210.216986] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9210.231456] LustreError: 323216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9212.229520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9212.533728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9216.185240] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9218.024914] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9222.183059] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9223.372645] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9228.018495] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9229.459811] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9233.384900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9233.385704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9233.397431] Lustre: Skipped 1 previous similar message [ 9233.408634] Lustre: Skipped 1 previous similar message [ 9239.518961] Lustre: server umount lustre-OST0000 complete [ 9243.106515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9243.114413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9248.694827] Lustre: server umount lustre-MDT0000 complete [ 9248.738576] LustreError: 324978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9248.753275] LustreError: 324978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9251.515268] LustreError: 321362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472210 with bad export cookie 186700505358031348 [ 9251.521143] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9251.528644] LustreError: 321362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9251.755449] Lustre: server umount lustre-MDT0001 complete [ 9259.791355] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 9261.947761] Key type lgssc unregistered [ 9262.227753] LNet: 327643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9262.232447] LNetError: 327643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9262.244443] LNet: Removed LNI 192.168.203.123@tcp [ 9262.896763] Key type .llcrypt unregistered [ 9262.899374] Key type ._llcrypt unregistered [ 9281.943901] Key type ._llcrypt registered [ 9281.949511] Key type .llcrypt registered [ 9282.084689] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9282.913353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9282.979066] alg: No test for adler32 (adler32-zlib) [ 9284.079839] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 9284.289763] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 9285.975292] Key type lgssc registered [ 9287.186098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9295.244406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9295.258481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9296.573618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9300.650828] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9306.082117] LustreError: 329036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9306.101171] LustreError: 329036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9306.705976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9307.036484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9309.950024] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9315.407498] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9319.920504] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9326.564561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9326.570718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9326.587848] Lustre: Skipped 1 previous similar message [ 9326.596434] Lustre: Skipped 1 previous similar message [ 9328.819042] Lustre: server umount lustre-MDT0000 complete [ 9331.680603] LustreError: 329036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9331.692275] LustreError: 329036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9332.005314] LustreError: 329018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472291 with bad export cookie 11957081624481557197 [ 9332.013124] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9332.014748] LustreError: 329018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9332.284045] Lustre: server umount lustre-MDT0001 complete [ 9344.156519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9344.354933] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9344.391632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9344.523606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9348.161265] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9356.625043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9356.719969] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9356.769030] Lustre: 331623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9357.126273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9361.373650] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9369.260710] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9373.972391] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9383.344264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9383.596482] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9383.613806] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9383.620904] LustreError: 333508:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 9383.630771] LustreError: 333508:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9383.635770] LustreError: 333508:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9383.709501] Lustre: server umount lustre-OST0000 complete [ 9383.713266] LustreError: 333508:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9393.385895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9393.567574] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9393.739054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9399.381888] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9400.925202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:289) [ 9406.293363] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9413.667739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9413.843310] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9413.948185] Lustre: lustre-OST0001: new disk, initializing [ 9413.952694] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9413.960123] Lustre: 335048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9414.113294] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9420.142432] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9423.930196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9423.939803] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9424.013211] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9425.508417] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9429.478351] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9429.486843] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9429.491464] Lustre: Skipped 1 previous similar message [ 9429.501053] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9429.508731] Lustre: Skipped 1 previous similar message [ 9433.843384] Lustre: server umount lustre-OST0001 complete [ 9434.622066] LustreError: 333976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9434.654478] LustreError: 333976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9439.718300] LustreError: 334428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9439.719707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9439.726799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9439.750345] LustreError: 334428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9439.762981] Lustre: Skipped 1 previous similar message [ 9443.225324] Lustre: server umount lustre-OST0000 complete [ 9449.440520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9449.446248] LustreError: Skipped 1 previous similar message [ 9449.450122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9449.467134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9449.472258] Lustre: Skipped 1 previous similar message [ 9452.554184] Lustre: server umount lustre-MDT0000 complete [ 9455.073645] LustreError: 331633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9455.096409] LustreError: 331633:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9455.429992] LustreError: 331616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472414 with bad export cookie 11957081624481557750 [ 9455.441216] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9455.446345] LustreError: 331616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9455.798211] Lustre: server umount lustre-MDT0001 complete [ 9463.979391] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 9466.288151] Key type lgssc unregistered [ 9466.577107] LNet: 337004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9466.590429] LNetError: 337004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9466.605707] LNet: Removed LNI 192.168.203.123@tcp [ 9467.449418] Key type .llcrypt unregistered [ 9467.451498] Key type ._llcrypt unregistered [ 9529.282254] Key type ._llcrypt registered [ 9529.284335] Key type .llcrypt registered [ 9529.385247] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 9543.971562] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9544.899915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9544.967211] alg: No test for adler32 (adler32-zlib) [ 9546.013943] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 9546.245677] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 9547.911365] Key type lgssc registered [ 9549.364113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9581.827933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9581.865593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9583.175374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9583.237592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9583.313422] Lustre: lustre-MDT0000: new disk, initializing [ 9583.383968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9583.400409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9588.937221] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9601.476639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9601.702554] Lustre: 341932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9601.787553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9601.790867] Lustre: Skipped 1 previous similar message [ 9601.847426] Lustre: lustre-MDT0001: new disk, initializing [ 9601.935303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9601.967605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9601.974427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9606.713289] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9615.150510] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9619.882741] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9625.296281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9625.529460] Lustre: lustre-OST0000: new disk, initializing [ 9625.532441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9625.537304] Lustre: 343901:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9625.619720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9630.591266] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9631.287024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9631.298927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9631.399690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9638.042996] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9641.787715] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9641.922506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9645.066450] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9645.229832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9647.073799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9647.080778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9647.086346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9647.090694] Lustre: Skipped 1 previous similar message [ 9647.101552] Lustre: Skipped 1 previous similar message [ 9652.194690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9652.203405] Lustre: Skipped 1 previous similar message [ 9652.908750] Lustre: server umount lustre-OST0000 complete [ 9657.317924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9657.327526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9657.337062] Lustre: Skipped 1 previous similar message [ 9662.281272] Lustre: server umount lustre-MDT0000 complete [ 9665.800234] LustreError: 342727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472624 with bad export cookie 13711156540460751289 [ 9665.801991] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9665.810555] LustreError: 342727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9666.125444] Lustre: server umount lustre-MDT0001 complete [ 9673.176784] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 17:57:11 (1788472631) [ 9679.570500] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 17:57:17 (1788472637) [ 9686.984072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9687.350219] LustreError: 346655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9687.374497] LustreError: 346655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9687.432787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9691.248512] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9692.649204] LustreError: 346656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9697.760763] LustreError: 346655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9698.183586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9698.491879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9702.325449] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9707.936066] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9712.353474] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9716.792805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9717.132049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9721.858843] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9727.486738] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9739.475476] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9740.968720] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9742.263297] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9743.523978] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9745.019428] Lustre: Failing over lustre-MDT0000 [ 9745.327892] Lustre: server umount lustre-MDT0000 complete [ 9747.937889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9747.943779] LustreError: 346650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9747.945468] Lustre: Skipped 2 previous similar messages [ 9747.957737] LustreError: 346650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9753.058143] LustreError: 346656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9753.068559] LustreError: 346656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9762.810101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9762.930249] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9763.172724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9763.217328] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9764.706943] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9766.600502] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9768.428731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9768.446065] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9768.514464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9775.045090] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9776.434362] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9779.644317] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9781.333353] Lustre: Failing over lustre-MDT0000 [ 9781.625491] Lustre: server umount lustre-MDT0000 complete [ 9783.777410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9783.777722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9783.781847] LustreError: Skipped 1 previous similar message [ 9783.788766] LustreError: 346650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9783.796799] LustreError: 346650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9799.141451] Lustre: 340153:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788472742/real 1788472742] req@000000002c14a9d9 x1875349345486080/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788472758 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9799.179666] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9800.544847] LustreError: 348494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9800.560630] LustreError: 348494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 9800.846821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9809.379697] LustreError: 340151:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000084b116cc x1875349345490816/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9809.622581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9809.674248] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9809.784740] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9813.751989] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9815.037292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9815.046662] Lustre: Skipped 2 previous similar messages [ 9815.085086] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 9815.133978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9821.695106] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9822.845564] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9830.376924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9830.387153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9830.393208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9830.401532] Lustre: Skipped 3 previous similar messages [ 9830.420538] Lustre: Skipped 3 previous similar messages [ 9833.884637] Lustre: server umount lustre-MDT0000 complete [ 9835.493386] LustreError: 346656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9835.519117] LustreError: 346656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 9837.148211] LustreError: 349446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472796 with bad export cookie 13711156540460754012 [ 9837.150727] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9837.162443] LustreError: 349446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9837.483527] Lustre: server umount lustre-MDT0001 complete [ 9851.286937] Lustre: server umount lustre-OST0000 complete [ 9865.740681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9870.882765] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9877.939371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9878.237457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9878.243588] Lustre: Skipped 1 previous similar message [ 9881.923967] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9887.830842] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9892.644216] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9898.699319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9905.087323] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9911.292095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9911.332752] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9916.386814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9916.403920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9916.408348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9916.417748] Lustre: Skipped 2 previous similar messages [ 9916.429889] Lustre: Skipped 1 previous similar message [ 9917.806679] Lustre: server umount lustre-OST0000 complete [ 9926.629307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9926.636052] Lustre: Skipped 2 previous similar messages [ 9927.494135] Lustre: server umount lustre-MDT0000 complete [ 9930.603914] LustreError: 352995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472889 with bad export cookie 13711156540460754754 [ 9930.604495] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9930.628154] LustreError: 352995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9930.955488] Lustre: server umount lustre-MDT0001 complete [ 9940.375642] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 9943.023642] Key type lgssc unregistered [ 9943.241856] LNet: 356556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9943.253634] LNetError: 356556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9943.270191] LNet: Removed LNI 192.168.203.123@tcp [ 9943.905612] Key type .llcrypt unregistered [ 9943.908354] Key type ._llcrypt unregistered [ 9966.269778] Key type ._llcrypt registered [ 9966.271479] Key type .llcrypt registered [ 9966.357396] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 9967.074218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9967.172067] alg: No test for adler32 (adler32-zlib) [ 9968.214339] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 9968.448930] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 9970.127435] Key type lgssc registered [ 9971.234794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9977.745953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9977.774562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9979.070445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9983.427408] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 9989.606088] LustreError: 358067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9989.629037] LustreError: 358067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9991.176133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9991.428838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9996.686459] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10004.015528] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10009.719047] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10016.360946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10017.048179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10023.517221] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10024.239293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10031.142103] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10046.946127] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10046.951053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10046.964023] Lustre: Skipped 1 previous similar message [10051.272162] Lustre: server umount lustre-MDT0000 complete [10052.067226] LustreError: 358066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10052.084229] LustreError: 358066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10054.151334] LustreError: 358047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473013 with bad export cookie 1223905400790369462 [10054.162337] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10054.168802] LustreError: 358047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10054.485749] Lustre: server umount lustre-MDT0001 complete [10067.816900] Lustre: server umount lustre-OST0000 complete [10077.304815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10077.709445] LustreError: 361795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10077.773619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10080.983537] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10082.784967] LustreError: 361796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10086.887279] LustreError: 361795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10087.135401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10087.468362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10090.463364] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10095.742207] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10099.920390] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10105.167817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10105.470871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10111.761774] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10119.298860] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10122.885455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10138.594379] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10138.596725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10138.597775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10138.597780] Lustre: Skipped 2 previous similar messages [10138.602951] Lustre: Skipped 2 previous similar messages [10141.547700] Lustre: server umount lustre-MDT0000 complete [10143.724813] LustreError: 361791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10143.738109] LustreError: 361791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10145.175172] LustreError: 361777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473104 with bad export cookie 1223905400790370575 [10145.177513] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10145.180536] LustreError: 361777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10145.427427] Lustre: server umount lustre-MDT0001 complete [10158.755274] Lustre: server umount lustre-OST0000 complete [10174.125590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10174.541217] LustreError: 365707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10174.617184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10178.232076] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10184.736645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10189.651379] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10197.295201] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10202.409989] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10210.489175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10210.746565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10210.757235] Lustre: Skipped 1 previous similar message [10216.213614] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10218.924796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10222.583501] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10227.168455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10227.175449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10227.190548] Lustre: Skipped 1 previous similar message [10227.196206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10227.208453] Lustre: Skipped 2 previous similar messages [10229.217946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10229.222453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10229.237615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10229.271823] Lustre: Skipped 1 previous similar message [10232.608777] Lustre: server umount lustre-OST0000 complete [10236.387512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10236.395312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10236.423728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10242.296245] Lustre: server umount lustre-MDT0000 complete [10244.577764] LustreError: 365708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10244.607856] LustreError: 365708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10245.730594] LustreError: 365687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473204 with bad export cookie 1223905400790371597 [10245.734547] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10245.752031] LustreError: 365687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10246.052309] Lustre: server umount lustre-MDT0001 complete [10255.032735] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [10257.564620] Key type lgssc unregistered [10257.855358] LNet: 369237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10257.865464] LNetError: 369237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10258.926549] LNet: Removed LNI 192.168.203.123@tcp [10259.718267] Key type .llcrypt unregistered [10259.722247] Key type ._llcrypt unregistered [10281.776947] Key type ._llcrypt registered [10281.779240] Key type .llcrypt registered [10281.870783] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [10282.836461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10282.858804] alg: No test for adler32 (adler32-zlib) [10283.936313] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [10284.188683] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [10285.895346] Key type lgssc registered [10286.829650] Lustre: Echo OBD driver; http://www.lustre.org/ [10294.282493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10294.309521] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10294.331194] LustreError: 370670:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10294.358706] LustreError: 370670:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10294.373671] LustreError: 370670:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10294.379133] LustreError: 370670:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10294.386882] LustreError: 370670:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10301.323782] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10301.336552] LustreError: 371066:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10301.365352] LustreError: 371066:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10301.373550] LustreError: 371066:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10301.394796] LustreError: 371066:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10301.411485] LustreError: 371066:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10323.358691] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [10325.832190] Key type lgssc unregistered [10326.055563] LNet: 372069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10326.060593] LNetError: 372069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10326.089054] LNet: Removed LNI 192.168.203.123@tcp [10326.965303] Key type .llcrypt unregistered [10326.969166] Key type ._llcrypt unregistered [10342.206646] Key type ._llcrypt registered [10342.209548] Key type .llcrypt registered [10342.320402] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [10354.307849] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [10355.392761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10355.427186] alg: No test for adler32 (adler32-zlib) [10356.441750] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [10356.674466] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [10358.327149] Key type lgssc registered [10359.357459] Lustre: Echo OBD driver; http://www.lustre.org/ [10394.709150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10394.724147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10396.064927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10396.087371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10396.253284] Lustre: lustre-MDT0000: new disk, initializing [10396.329107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10396.356847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10401.584246] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10410.994740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10411.109944] Lustre: 375201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10411.157249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10411.160907] Lustre: Skipped 1 previous similar message [10411.209694] Lustre: lustre-MDT0001: new disk, initializing [10411.288072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10411.360008] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10411.372570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10415.379822] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10425.573301] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10431.549992] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10438.516117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10438.784841] Lustre: lustre-OST0000: new disk, initializing [10438.788623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10438.810865] Lustre: 377173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10438.939884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10445.453302] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10448.447713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10448.457775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10448.558881] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10454.223558] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10459.887875] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10460.085249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10464.758494] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10465.071844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10469.344166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10469.346109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10469.356800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10469.373353] Lustre: Skipped 1 previous similar message [10469.392298] Lustre: Skipped 1 previous similar message [10473.280041] Lustre: server umount lustre-OST0000 complete [10478.048801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10478.076376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10478.108898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10479.586381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10479.588391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10484.704915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10484.712902] Lustre: Skipped 2 previous similar messages [10489.825461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10489.832738] Lustre: Skipped 1 previous similar message [10491.359887] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10491.614983] Lustre: server umount lustre-MDT0000 complete [10495.258578] LustreError: 375193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473454 with bad export cookie 10704281635038377066 [10495.263510] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10495.277202] LustreError: 375193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10495.767905] Lustre: server umount lustre-MDT0001 complete [10504.346909] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10506.190872] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:11:03 (1788473463) [10513.296949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10513.861509] LustreError: 379632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10513.889468] LustreError: 379632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10513.970788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10518.173881] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10519.009592] LustreError: 379633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10524.135276] LustreError: 379632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10525.899526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10526.267493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10530.718728] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10538.774243] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10544.979535] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10552.716079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10552.999391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10559.465582] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10565.869727] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10571.892153] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10572.076980] Lustre: lustre-OST0001: new disk, initializing [10572.080892] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10572.090188] Lustre: 382467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10572.192622] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10577.696173] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10579.573952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10579.590601] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10579.686583] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10585.933513] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10590.181484] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10590.186900] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10590.205567] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10590.213834] Lustre: Skipped 1 previous similar message [10595.527433] Lustre: server umount lustre-OST0001 complete [10597.730610] LustreError: 381454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10605.546431] LustreError: 381454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10605.571993] LustreError: 381454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10612.704641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10612.710654] LustreError: Skipped 1 previous similar message [10612.713859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10612.739872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10612.748968] Lustre: Skipped 4 previous similar messages [10615.778453] LustreError: 383392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10615.803498] LustreError: 383392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10618.018898] Lustre: server umount lustre-OST0000 complete [10623.459254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10623.465833] LustreError: Skipped 1 previous similar message [10623.477165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10623.487987] Lustre: Skipped 1 previous similar message [10627.673756] Lustre: server umount lustre-MDT0000 complete [10631.365408] LustreError: 379613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473590 with bad export cookie 10704281635038377948 [10631.370597] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10631.376456] LustreError: 379613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10637.715734] Lustre: server umount lustre-MDT0001 complete [10646.890171] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [10650.061304] Key type lgssc unregistered [10650.383049] LNet: 384450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10650.395689] LNetError: 384450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10650.418357] LNet: Removed LNI 192.168.203.123@tcp [10651.317208] Key type .llcrypt unregistered [10651.319920] Key type ._llcrypt unregistered [10698.439236] Key type ._llcrypt registered [10698.441162] Key type .llcrypt registered [10698.533951] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [10699.443294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10699.453470] alg: No test for adler32 (adler32-zlib) [10700.432352] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [10700.586221] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [10702.263196] Key type lgssc registered [10703.275206] Lustre: Echo OBD driver; http://www.lustre.org/ [10710.270496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10710.288475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10711.584445] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10711.615857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10711.749809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10715.880462] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10724.544615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10724.637557] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10724.659717] Lustre: 387244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10724.950523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10728.538000] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10735.088939] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10740.984199] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10745.313052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10745.317465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10745.329110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10747.360575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10747.362033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10749.978664] Lustre: server umount lustre-MDT0000 complete [10754.209773] LustreError: 387236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473713 with bad export cookie 9693215841138228190 [10754.218969] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10754.637781] Lustre: server umount lustre-MDT0001 complete [10765.887360] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:15:23 (1788473723) [10772.209516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10772.597377] LustreError: 389757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10772.613550] LustreError: 389757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10772.658596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10776.142545] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10778.081507] LustreError: 389758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10783.216207] LustreError: 389757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10784.070602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10784.348349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10789.016268] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10797.804875] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10805.030544] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10812.126726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10812.479598] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10812.735269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10818.679899] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10827.294160] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10834.378647] Lustre: Permanently deactivating lustre-OST0000 [10834.403184] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10842.158255] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10844.212729] LustreError: 392742:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10844.221632] LustreError: 392742:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10844.229699] LustreError: 392742:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10847.912567] LustreError: 392791:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10847.928114] LustreError: 392791:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10855.883485] Lustre: server umount lustre-OST0000 complete [10858.718197] LustreError: 393040:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10858.723830] LustreError: 393040:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10861.024292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10861.035658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10861.049447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10861.055450] Lustre: Skipped 1 previous similar message [10863.074210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10863.075451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10868.193361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10868.207309] Lustre: Skipped 2 previous similar messages [10870.426912] Lustre: server umount lustre-MDT0000 complete [10873.317120] LustreError: 390951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10873.336824] LustreError: 390951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10874.633370] LustreError: 389737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473833 with bad export cookie 9693215841138228771 [10874.636739] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10874.643599] LustreError: 389737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10884.303277] Lustre: server umount lustre-MDT0001 complete [10892.634437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10898.052933] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10901.099156] LustreError: 394200:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10907.418323] Lustre: 394539:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.123@tcp [10909.416363] Lustre: server umount MGS complete [10919.754630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10920.231834] LustreError: 395264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10920.244979] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10920.255270] Lustre: Skipped 1 previous similar message [10920.303770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10924.035820] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10930.660614] LustreError: 395265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10930.672818] LustreError: 395265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10931.414270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10931.702986] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10936.250481] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10943.335372] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10948.247211] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10949.954144] Lustre: Permanently reactivating lustre-OST0000 [10949.979305] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10949.987928] Lustre: Skipped 2 previous similar messages [10977.247366] LustreError: 397043:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10983.774917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10984.023331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10984.029982] Lustre: Skipped 1 previous similar message [10989.579950] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [10997.504626] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11002.852861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11002.854303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11002.860689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11002.883316] Lustre: Skipped 1 previous similar message [11004.749728] Lustre: server umount lustre-OST0000 complete [11013.091224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11013.106761] Lustre: Skipped 1 previous similar message [11015.337778] Lustre: server umount lustre-MDT0000 complete [11018.189835] LustreError: 396835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473977 with bad export cookie 9693215841138229821 [11018.204084] LustreError: 396835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11018.209283] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11018.447354] Lustre: server umount lustre-MDT0001 complete [11026.384415] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [11029.483621] Key type lgssc unregistered [11029.842569] LNet: 399626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11029.855686] LNetError: 399626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11029.875106] LNet: Removed LNI 192.168.203.123@tcp [11030.866323] Key type .llcrypt unregistered [11030.872056] Key type ._llcrypt unregistered [11049.847440] Key type ._llcrypt registered [11049.849300] Key type .llcrypt registered [11049.920812] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [11063.197743] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [11063.672691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11063.681392] alg: No test for adler32 (adler32-zlib) [11064.659822] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [11064.829690] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [11066.463276] Key type lgssc registered [11067.383190] Lustre: Echo OBD driver; http://www.lustre.org/ [11099.135360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11099.202393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11100.572314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11100.617686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11100.715463] Lustre: lustre-MDT0000: new disk, initializing [11100.814981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11100.846393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11105.599470] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11114.735634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11114.833832] Lustre: 402777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11114.889261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11114.894457] Lustre: Skipped 1 previous similar message [11114.965246] Lustre: lustre-MDT0001: new disk, initializing [11115.042710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11115.076233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11115.086596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11119.388278] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11129.518881] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11134.099382] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11140.587503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11141.074946] Lustre: lustre-OST0000: new disk, initializing [11141.079379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11141.090243] Lustre: 404746:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11141.260428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11148.407035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11148.426330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11148.587393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11149.414710] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11160.298031] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11164.379072] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11164.641935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11168.913496] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11169.157518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11174.368412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11174.380766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11174.396205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11174.404902] Lustre: Skipped 1 previous similar message [11179.489164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11179.498179] Lustre: Skipped 1 previous similar message [11184.613075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11184.620068] Lustre: Skipped 1 previous similar message [11185.119181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11185.320162] Lustre: server umount lustre-OST0000 complete [11189.729336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11189.747043] Lustre: Skipped 1 previous similar message [11189.756594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11189.766272] Lustre: Skipped 1 previous similar message [11194.851088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11194.858402] Lustre: Skipped 1 previous similar message [11195.690971] Lustre: server umount lustre-MDT0000 complete [11199.379702] LustreError: 402770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474158 with bad export cookie 15760006776489283672 [11199.381944] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11199.394192] LustreError: 402770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11199.806853] Lustre: server umount lustre-MDT0001 complete [11207.764708] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11209.353627] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 18:22:47 (1788474167) [11215.005459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11215.429417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11220.331597] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11220.450180] LustreError: 407208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11220.465805] LustreError: 407208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11225.571521] LustreError: 407208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11229.087574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11229.421874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11234.108517] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11241.204880] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11245.582677] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11252.015350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11252.399547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11258.740290] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11265.234519] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11267.244927] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11269.603484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11275.232993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11275.238601] LustreError: Skipped 1 previous similar message [11275.243502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11275.257211] Lustre: Skipped 1 previous similar message [11275.265742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11276.256834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11280.534143] Lustre: server umount lustre-OST0000 complete [11285.985333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11285.996882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11286.005997] Lustre: Skipped 1 previous similar message [11289.603162] Lustre: server umount lustre-MDT0000 complete [11291.620111] LustreError: 407208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11291.649297] LustreError: 407208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11293.484160] LustreError: 407188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474252 with bad export cookie 15760006776489284554 [11293.491078] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11293.498608] LustreError: 407188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11293.871359] Lustre: server umount lustre-MDT0001 complete [11304.971148] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [11308.365350] Key type lgssc unregistered [11308.742551] LNet: 410933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11308.758635] LNetError: 410933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11308.779511] LNet: Removed LNI 192.168.203.123@tcp [11309.662937] Key type .llcrypt unregistered [11309.665132] Key type ._llcrypt unregistered [11333.251803] Key type ._llcrypt registered [11333.260841] Key type .llcrypt registered [11333.437401] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [11334.006293] Key type .llcrypt unregistered [11334.016422] Key type ._llcrypt unregistered [11355.722772] Key type ._llcrypt registered [11355.730590] Key type .llcrypt registered [11355.913717] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [11357.112996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11357.157150] alg: No test for adler32 (adler32-zlib) [11358.196280] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [11358.511659] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [11360.223175] Key type lgssc registered [11361.427423] Lustre: Echo OBD driver; http://www.lustre.org/ [11370.300173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11370.310651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11371.658214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11376.162448] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11381.217259] LustreError: 413141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11381.233978] LustreError: 413141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11384.254087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11384.623713] LustreError: 413142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11384.656020] LustreError: 413142:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11384.661924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11389.925325] LustreError: 413141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11389.968859] LustreError: 413141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11391.169904] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11395.042584] LustreError: 413142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11395.062864] LustreError: 413142:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11396.080085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11397.670816] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11401.188636] LustreError: 413141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11401.202110] LustreError: 413141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11403.464394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11403.723485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11407.196269] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11416.481470] Lustre: server umount lustre-OST0000 complete [11423.207194] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11423.220241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11423.234795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11424.230102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11424.230878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11429.345323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11429.364189] Lustre: Skipped 2 previous similar messages [11430.479814] Lustre: server umount lustre-MDT0000 complete [11433.669563] LustreError: 413123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474392 with bad export cookie 17975152950929730434 [11433.678174] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11433.682452] LustreError: 413123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11443.338385] Lustre: server umount lustre-MDT0001 complete [11453.831601] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [11458.046828] Key type lgssc unregistered [11458.534903] LNet: 416290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11458.542481] LNetError: 416290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11458.562124] LNet: Removed LNI 192.168.203.123@tcp [11459.832213] Key type .llcrypt unregistered [11459.839941] Key type ._llcrypt unregistered [11487.338283] Key type ._llcrypt registered [11487.340390] Key type .llcrypt registered [11487.420778] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [11488.131677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11488.161217] alg: No test for adler32 (adler32-zlib) [11489.181616] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [11489.330951] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [11490.967174] Key type lgssc registered [11491.715159] Lustre: Echo OBD driver; http://www.lustre.org/ [11498.753231] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11498.777935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11514.335346] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11519.455304] LustreError: 417678:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 4: rc = -110 [11546.143191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11551.304580] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11557.869147] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11563.508536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11564.031135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11564.100375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11568.188659] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11569.123566] LustreError: 417704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11569.131923] LustreError: 417704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11574.245785] LustreError: 417703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11575.552871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11575.829482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11579.320242] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11584.966867] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11589.225488] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11596.257937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11596.261299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11596.266276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11596.961181] Lustre: server umount lustre-OST0000 complete [11601.376375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11601.384823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11601.412574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11602.400756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11602.417981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11602.434158] Lustre: Skipped 1 previous similar message [11607.524828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11607.544686] Lustre: Skipped 1 previous similar message [11610.750040] Lustre: server umount lustre-MDT0000 complete [11612.640514] LustreError: 418719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11612.658130] LustreError: 418719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11614.745040] LustreError: 417686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474573 with bad export cookie 11265374832142880959 [11614.765191] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11615.105772] Lustre: server umount lustre-MDT0001 complete [11625.967426] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [11628.718380] Key type lgssc unregistered [11628.990713] LNet: 421244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11629.002320] LNetError: 421244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11629.033759] LNet: Removed LNI 192.168.203.123@tcp [11630.000326] Key type .llcrypt unregistered [11630.005493] Key type ._llcrypt unregistered [11653.076159] Key type ._llcrypt registered [11653.077798] Key type .llcrypt registered [11653.178200] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [11654.204753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11654.222453] alg: No test for adler32 (adler32-zlib) [11655.240700] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [11655.518115] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [11657.183216] Key type lgssc registered [11658.292657] Lustre: Echo OBD driver; http://www.lustre.org/ [11666.713787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11666.756508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11668.714788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11674.594850] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11678.194462] LustreError: 422655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11678.216313] LustreError: 422655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11683.310086] LustreError: 422656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11684.150555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11684.658787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11689.245574] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11696.299554] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11701.158058] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11706.335346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11706.658605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11711.358847] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11713.855633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11717.712959] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11724.258755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11724.262410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11724.298105] Lustre: Skipped 2 previous similar messages [11724.302863] Lustre: Skipped 1 previous similar message [11725.694414] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [11725.714204] Lustre: Skipped 1 previous similar message [11727.972680] Lustre: server umount lustre-MDT0000 complete [11729.379053] LustreError: 424326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11729.406812] LustreError: 424326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11730.414018] LustreError: 422635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.23@tcp arrived at 1788474689 with bad export cookie 12453866245031663223 [11732.365818] LustreError: 422636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474691 with bad export cookie 12453866245031662320 [11732.397901] LustreError: 422636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11732.408868] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11734.496982] LustreError: 423670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11734.509832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11734.518442] LustreError: 423670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11734.536501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11737.505189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11738.684898] Lustre: server umount lustre-OST0000 complete [11742.626711] LustreError: 423670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11742.646047] LustreError: 423670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11745.787972] Lustre: server umount lustre-MDT0001 complete [11756.537263] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [11759.324994] Key type lgssc unregistered [11759.623585] LNet: 426339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11759.630311] LNetError: 426339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11759.657098] LNet: Removed LNI 192.168.203.123@tcp [11760.426649] Key type .llcrypt unregistered [11760.431414] Key type ._llcrypt unregistered [11781.606731] Key type ._llcrypt registered [11781.609133] Key type .llcrypt registered [11781.723524] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [11782.467947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11782.478990] alg: No test for adler32 (adler32-zlib) [11783.507058] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [11783.753822] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [11785.399154] Key type lgssc registered [11786.429975] Lustre: Echo OBD driver; http://www.lustre.org/ [11793.225577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11793.246122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11794.649293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11798.678551] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11804.131651] LustreError: 427750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11804.150307] LustreError: 427750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11806.317437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11806.560760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11810.394495] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11816.507435] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11821.061235] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11826.725435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11827.046368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11832.794696] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11839.349622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11839.589528] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11844.585561] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11844.589336] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11844.606071] Lustre: Skipped 1 previous similar message [11847.550430] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [11847.557727] Lustre: Skipped 1 previous similar message [11849.699851] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11850.190255] Lustre: server umount lustre-MDT0001 complete [11854.818929] LustreError: 427750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11854.853194] LustreError: 427750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11859.937910] LustreError: 428477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11859.938533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11859.965419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11859.977863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11859.980796] Lustre: Skipped 1 previous similar message [11861.526492] Lustre: server umount lustre-OST0000 complete [11864.836183] Lustre: server umount lustre-MDT0000 complete [11874.361534] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [11876.539751] Key type lgssc unregistered [11876.792644] LNet: 431432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11876.805809] LNetError: 431432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11876.819721] LNet: Removed LNI 192.168.203.123@tcp [11877.534150] Key type .llcrypt unregistered [11877.535959] Key type ._llcrypt unregistered [11899.209685] Key type ._llcrypt registered [11899.212347] Key type .llcrypt registered [11899.315269] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [11914.180628] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [11915.259134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11915.290399] alg: No test for adler32 (adler32-zlib) [11916.620695] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [11916.861346] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [11918.503151] Key type lgssc registered [11919.675483] Lustre: Echo OBD driver; http://www.lustre.org/ [11955.130617] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [11970.249960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11970.291039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11971.557432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11971.577070] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11971.668514] Lustre: lustre-MDT0000: new disk, initializing [11971.759579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11971.775443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11976.078748] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [11991.417886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11991.695766] Lustre: lustre-OST0000: new disk, initializing [11991.703129] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11991.714769] Lustre: Skipped 1 previous similar message [11991.727104] Lustre: 436475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11991.786033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11997.858083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11997.877021] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11997.904475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11998.087461] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12006.459797] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12019.656294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12019.737592] Lustre: 435457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12019.810479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12019.921630] Lustre: lustre-MDT0001: new disk, initializing [12020.045697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12020.094175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12020.105714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12023.988467] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12031.596795] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12031.784865] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12035.438376] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12035.679364] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12040.580621] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [12043.754087] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12043.754679] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12043.770134] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12045.675825] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [12045.686578] Lustre: Skipped 1 previous similar message [12045.982385] Lustre: server umount lustre-MDT0001 complete [12053.994771] LustreError: 437391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12054.013156] LustreError: 437391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12059.111649] LustreError: 435467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12059.139108] LustreError: 435467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12064.233199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12064.247483] LustreError: 435468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12064.268511] Lustre: Skipped 1 previous similar message [12064.270171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12069.366403] LustreError: 435467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12069.411513] LustreError: 435467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12070.164304] Lustre: server umount lustre-OST0000 complete [12074.486877] LustreError: 435467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12082.288374] Lustre: server umount lustre-MDT0000 complete [12095.859795] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12098.424605] Key type lgssc unregistered [12098.696660] LNet: 440222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12098.705200] LNetError: 440222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12099.750304] LNet: Removed LNI 192.168.203.123@tcp [12100.587648] Key type .llcrypt unregistered [12100.589987] Key type ._llcrypt unregistered [12137.551940] Key type ._llcrypt registered [12137.554679] Key type .llcrypt registered [12137.660378] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [12138.022905] Key type .llcrypt unregistered [12138.025120] Key type ._llcrypt unregistered [12161.374536] Key type ._llcrypt registered [12161.376569] Key type .llcrypt registered [12161.511855] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [12162.630336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12162.661826] alg: No test for adler32 (adler32-zlib) [12163.840356] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [12164.107467] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [12165.847258] Key type lgssc registered [12167.267932] Lustre: Echo OBD driver; http://www.lustre.org/ [12196.240180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12196.264506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12197.594875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12197.673303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12197.841505] Lustre: lustre-MDT0000: new disk, initializing [12197.958506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12198.013809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12202.335251] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12212.209566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12212.335625] Lustre: 443593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12212.390983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12212.393112] Lustre: Skipped 1 previous similar message [12212.470630] Lustre: lustre-MDT0001: new disk, initializing [12212.559094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12212.602251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12212.620346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12216.063638] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12225.805070] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12230.902245] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12238.288221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12238.502372] Lustre: lustre-OST0000: new disk, initializing [12238.505689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12238.510589] Lustre: 445567:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12238.586751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12246.419826] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12247.621645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12247.626874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12247.746035] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12253.676122] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12263.392676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12263.403539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12263.420364] Lustre: Skipped 2 previous similar messages [12263.423277] Lustre: Skipped 1 previous similar message [12266.700453] Lustre: server umount lustre-MDT0000 complete [12269.754159] LustreError: 443586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475228 with bad export cookie 3793816459586288737 [12269.765633] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12269.773027] LustreError: 443586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12270.067818] Lustre: server umount lustre-MDT0001 complete [12283.864135] Lustre: server umount lustre-OST0000 complete [12293.545801] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 18:40:51 (1788475251) [12300.586831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12301.038167] LustreError: 447595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12301.056910] LustreError: 447595:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12301.101715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12304.911110] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12306.405952] LustreError: 447596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12311.521036] LustreError: 447595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12312.034900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12312.303162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12315.405297] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12322.989726] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12328.715681] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12350.640104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12350.827145] Lustre: Found index 0 for lustre-OST0000, updating log [12351.033405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12357.583045] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [12364.360560] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12372.527191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12382.691610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12382.692211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12382.695401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12382.695412] Lustre: Skipped 1 previous similar message [12382.697143] LustreError: Skipped 1 previous similar message [12382.724355] Lustre: Skipped 1 previous similar message [12383.780914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12383.793272] Lustre: Skipped 2 previous similar messages [12385.352984] Lustre: server umount lustre-OST0000 complete [12404.707840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12404.716269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12404.743788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12407.233916] Lustre: server umount lustre-MDT0000 complete [12410.341538] LustreError: 447595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12410.360720] LustreError: 447595:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12411.256206] LustreError: 447577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475370 with bad export cookie 3793816459586290130 [12411.263281] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12411.282982] LustreError: 447577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12411.659634] Lustre: server umount lustre-MDT0001 complete [12416.038396] LNet: 1 local NIs in recovery (showing 1): 192.168.203.123@tcp [12421.864743] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 18:42:59 (1788475379) [13038.198555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13038.617597] LustreError: 454028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13038.684847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13042.921610] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [13043.690263] LustreError: 454029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13048.805145] LustreError: 454028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13051.480116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13051.731814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13056.261588] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [13064.306072] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13070.402422] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13090.057763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13090.537860] Lustre: Found index 0 for lustre-OST0000, updating log [13090.969306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13092.960134] LNet: 1 local NIs in recovery (showing 1): 192.168.203.123@tcp [13098.098425] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [13106.140789] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13108.027885] Lustre: server umount lustre-OST0000 complete [13112.099165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13112.111636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13112.111685] Lustre: Skipped 2 previous similar messages [13112.125979] Lustre: Skipped 3 previous similar messages [13113.129397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13113.146975] Lustre: Skipped 1 previous similar message [13114.150046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13114.157955] Lustre: Skipped 1 previous similar message [13116.197059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13116.210374] Lustre: Skipped 3 previous similar messages [13117.322943] Lustre: server umount lustre-MDT0000 complete [13118.242326] LustreError: 454023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13118.263098] LustreError: 454023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13120.207541] LustreError: 454763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476079 with bad export cookie 3793816459586291215 [13120.209284] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13120.220121] LustreError: 454763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13120.490595] Lustre: server umount lustre-MDT0001 complete [13147.691324] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [13153.375608] LNet: 1 local NIs in recovery (showing 1): 192.168.203.123@tcp [13155.845162] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [13187.529474] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:55:45 (1788476145) [13203.069061] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [13205.961424] Key type lgssc unregistered [13206.251651] LNet: 460635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13206.255838] LNetError: 460635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13301.542202] LNet: Removed LNI 192.168.203.123@tcp [13302.409300] Key type .llcrypt unregistered [13302.414033] Key type ._llcrypt unregistered [13320.071936] Key type ._llcrypt registered [13320.074965] Key type .llcrypt registered [13320.180683] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [13321.243626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13321.326196] alg: No test for adler32 (adler32-zlib) [13322.376697] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [13322.637928] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [13324.311131] Key type lgssc registered [13325.339145] Lustre: Echo OBD driver; http://www.lustre.org/ [13329.518549] LNet: Added LNI 192.168.203.123@tcp100 [8/256/0/180] [13330.246224] LNet: Added LNI 192.168.203.123@tcp101 [8/256/0/180] [13330.930202] LNet: Added LNI 192.168.203.123@tcp102 [8/256/0/180] [13331.688058] LNet: Added LNI 192.168.203.123@tcp103 [8/256/0/180] [13332.472477] LNet: Added LNI 192.168.203.123@tcp104 [8/256/0/180] [13333.380534] LNet: Added LNI 192.168.203.123@tcp105 [8/256/0/180] [13334.138936] LNet: Added LNI 192.168.203.123@tcp106 [8/256/0/180] [13335.107199] LNet: Added LNI 192.168.203.123@tcp107 [8/256/0/180] [13335.749859] LNet: Added LNI 192.168.203.123@tcp108 [8/256/0/180] [13336.499427] LNet: Added LNI 192.168.203.123@tcp109 [8/256/0/180] [13337.216918] LNet: Added LNI 192.168.203.123@tcp110 [8/256/0/180] [13338.047778] LNet: Added LNI 192.168.203.123@tcp111 [8/256/0/180] [13338.749568] LNet: Added LNI 192.168.203.123@tcp112 [8/256/0/180] [13339.478195] LNet: Added LNI 192.168.203.123@tcp113 [8/256/0/180] [13340.298711] LNet: Added LNI 192.168.203.123@tcp114 [8/256/0/180] [13341.193184] LNet: Added LNI 192.168.203.123@tcp115 [8/256/0/180] [13342.127774] LNet: Added LNI 192.168.203.123@tcp116 [8/256/0/180] [13343.163585] LNet: Added LNI 192.168.203.123@tcp117 [8/256/0/180] [13343.807265] LNet: Added LNI 192.168.203.123@tcp118 [8/256/0/180] [13344.792653] LNet: Added LNI 192.168.203.123@tcp119 [8/256/0/180] [13345.653927] LNet: Added LNI 192.168.203.123@tcp120 [8/256/0/180] [13346.647263] LNet: Added LNI 192.168.203.123@tcp121 [8/256/0/180] [13347.513749] LNet: Added LNI 192.168.203.123@tcp122 [8/256/0/180] [13348.530321] LNet: Added LNI 192.168.203.123@tcp123 [8/256/0/180] [13349.485340] LNet: Added LNI 192.168.203.123@tcp124 [8/256/0/180] [13350.411494] LNet: Added LNI 192.168.203.123@tcp125 [8/256/0/180] [13351.389984] LNet: Added LNI 192.168.203.123@tcp126 [8/256/0/180] [13352.776980] LNet: Added LNI 192.168.203.123@tcp127 [8/256/0/180] [13354.149657] LNet: Added LNI 192.168.203.123@tcp128 [8/256/0/180] [13355.218225] LNet: Added LNI 192.168.203.123@tcp129 [8/256/0/180] [13356.364858] LNet: Added LNI 192.168.203.123@tcp130 [8/256/0/180] [13357.525069] LNet: Added LNI 192.168.203.123@tcp131 [8/256/0/180] [13358.622658] LNet: Added LNI 192.168.203.123@tcp132 [8/256/0/180] [13359.816470] LNet: Added LNI 192.168.203.123@tcp133 [8/256/0/180] [13360.790335] LNet: Added LNI 192.168.203.123@tcp134 [8/256/0/180] [13362.087620] LNet: Added LNI 192.168.203.123@tcp135 [8/256/0/180] [13368.700369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13368.729543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13369.877877] Lustre: 463662:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.123@tcp: can't add NID 192.168.203.123@tcp130: rc = -75 [13369.953448] LNetError: 461120:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.123 [13369.969334] LNetError: Protocol error connecting to 192.168.203.123@tcp114 at host 192.168.203.123:988: is it running a compatible version of Lustre? [13369.988351] LNetError: Skipped 2 previous similar messages [13370.081843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13370.148366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13370.216598] Lustre: lustre-MDT0000: new disk, initializing [13370.278628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13370.292453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13370.847256] LNet: 7 local NIs in recovery (showing 7): 192.168.203.123@tcp124, 192.168.203.123@tcp125, 192.168.203.123@tcp127, 192.168.203.123@tcp128, 192.168.203.123@tcp129, 192.168.203.123@tcp123, 192.168.203.123@tcp126 [13370.848953] LNetError: 463692:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.123 [13370.876880] LNetError: 463692:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [13370.886455] LNetError: Protocol error connecting to 192.168.203.123@tcp112 at host 192.168.203.123:988: is it running a compatible version of Lustre? [13370.911666] LNetError: Skipped 7 previous similar messages [13372.896809] LNetError: 461118:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.123 [13372.914889] LNetError: 461118:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [13372.921667] LNetError: Protocol error connecting to 192.168.203.123@tcp126 at host 192.168.203.123:988: is it running a compatible version of Lustre? [13374.751353] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [13376.993070] LNetError: 461119:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.123 [13377.012869] LNetError: Protocol error connecting to 192.168.203.123@tcp126 at host 192.168.203.123:988: is it running a compatible version of Lustre? [13384.823739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13385.029491] Lustre: 463679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13385.151500] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13385.157899] Lustre: Skipped 1 previous similar message [13385.269310] Lustre: lustre-MDT0001: new disk, initializing [13385.323261] Lustre: 464481:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.123@tcp: can't add NID 192.168.203.123@tcp130: rc = -75 [13385.329782] Lustre: 464481:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13385.484849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13385.544663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13385.562885] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13389.979551] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [13391.931876] Lustre: 464845:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.123@tcp: can't add NID 192.168.203.123@tcp130: rc = -75 [13391.946046] Lustre: 464845:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13399.805871] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13400.481872] LNetError: 463751:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.123 [13400.496433] LNetError: Protocol error connecting to 192.168.203.123@tcp112 at host 192.168.203.123:988: is it running a compatible version of Lustre? [13405.330876] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13408.675475] LNetError: 463692:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.123 [13408.698882] LNetError: 463692:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [13408.710683] LNetError: Protocol error connecting to 192.168.203.123@tcp129 at host 192.168.203.123:988: is it running a compatible version of Lustre? [13408.728980] LNetError: Skipped 2 previous similar messages [13412.351675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13412.670920] Lustre: lustre-OST0000: new disk, initializing [13412.677755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13412.693392] Lustre: 465654:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13412.811426] Lustre: 465659:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.123@tcp: can't add NID 192.168.203.123@tcp130: rc = -75 [13412.821175] Lustre: 465659:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13412.883542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13419.030981] Lustre: 465967:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.123@tcp: can't add NID 192.168.203.123@tcp130: rc = -75 [13419.048519] Lustre: 465967:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13419.108981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13419.121842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13419.248834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13419.393885] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [13425.062459] LNetError: 461121:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.123 [13425.077988] LNetError: 461121:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [13425.091879] LNetError: Protocol error connecting to 192.168.203.123@tcp112 at host 192.168.203.123:988: is it running a compatible version of Lustre? [13425.110521] LNetError: Skipped 7 previous similar messages [13428.952151] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13431.263960] LNet: 1 local NIs in recovery (showing 1): 192.168.203.123@tcp126 [13439.968894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13439.977453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13439.978489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13439.991071] Lustre: Skipped 1 previous similar message [13445.095295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13445.108379] Lustre: Skipped 1 previous similar message [13450.207137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13450.215716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13450.231157] Lustre: Skipped 1 previous similar message [13450.375908] Lustre: server umount lustre-OST0000 complete [13455.337348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13455.346401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13455.348589] Lustre: Skipped 1 previous similar message [13455.353130] Lustre: Skipped 1 previous similar message [13460.663113] Lustre: server umount lustre-MDT0000 complete [13464.298235] LustreError: 463671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476423 with bad export cookie 6015433001746724184 [13464.303827] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13464.320658] LustreError: 463671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13464.609629] Lustre: server umount lustre-MDT0001 complete [13467.040151] LNetError: 463685:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.123 [13467.061712] LNetError: 463685:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [13467.070587] LNetError: Protocol error connecting to 192.168.203.123@tcp128 at host 192.168.203.123:988: is it running a compatible version of Lustre? [13467.084630] LNetError: Skipped 10 previous similar messages [13474.730615] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [13477.283820] Key type lgssc unregistered [13477.592344] LNet: 467565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13477.601880] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13477.607689] LNet: Removed LNI 192.168.203.123@tcp [13478.626430] LNet: Removed LNI 192.168.203.123@tcp100 [13478.630668] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13478.639667] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13478.659383] LNet: Removed LNI 192.168.203.123@tcp101 [13479.711203] LNet: Removed LNI 192.168.203.123@tcp102 [13479.714600] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13479.722063] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13480.735177] LNet: Removed LNI 192.168.203.123@tcp103 [13481.759156] LNet: Removed LNI 192.168.203.123@tcp104 [13481.763752] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13481.769830] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13481.778771] LNet: Removed LNI 192.168.203.123@tcp105 [13482.847180] LNet: Removed LNI 192.168.203.123@tcp106 [13483.871164] LNet: Removed LNI 192.168.203.123@tcp107 [13484.896169] LNet: Removed LNI 192.168.203.123@tcp108 [13485.919195] LNet: Removed LNI 192.168.203.123@tcp109 [13485.925595] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13485.934140] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13485.944118] LNet: Removed LNI 192.168.203.123@tcp110 [13487.007702] LNet: Removed LNI 192.168.203.123@tcp111 [13488.031532] LNet: Removed LNI 192.168.203.123@tcp112 [13489.060139] LNet: Removed LNI 192.168.203.123@tcp113 [13490.079482] LNet: Removed LNI 192.168.203.123@tcp114 [13491.103154] LNet: Removed LNI 192.168.203.123@tcp115 [13492.127696] LNet: Removed LNI 192.168.203.123@tcp116 [13493.151205] LNet: Removed LNI 192.168.203.123@tcp117 [13494.175188] LNet: Removed LNI 192.168.203.123@tcp118 [13494.178586] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13494.183867] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13494.192504] LNet: Removed LNI 192.168.203.123@tcp119 [13495.207640] LNet: Removed LNI 192.168.203.123@tcp120 [13496.225393] LNet: Removed LNI 192.168.203.123@tcp121 [13497.247850] LNet: Removed LNI 192.168.203.123@tcp122 [13498.271262] LNet: Removed LNI 192.168.203.123@tcp123 [13499.295328] LNet: Removed LNI 192.168.203.123@tcp124 [13500.319211] LNet: Removed LNI 192.168.203.123@tcp125 [13500.322108] LNet: Removed LNI 192.168.203.123@tcp126 [13501.348989] LNet: Removed LNI 192.168.203.123@tcp127 [13502.367145] LNet: Removed LNI 192.168.203.123@tcp128 [13503.391108] LNet: Removed LNI 192.168.203.123@tcp129 [13503.393208] LNet: Removed LNI 192.168.203.123@tcp130 [13503.403610] LNet: Removed LNI 192.168.203.123@tcp131 [13503.413909] LNet: Removed LNI 192.168.203.123@tcp132 [13503.427886] LNet: Removed LNI 192.168.203.123@tcp133 [13503.434123] LNet: Removed LNI 192.168.203.123@tcp134 [13503.450865] LNet: Removed LNI 192.168.203.123@tcp135 [13504.279243] Key type .llcrypt unregistered [13504.280749] Key type ._llcrypt unregistered [13529.726564] Key type ._llcrypt registered [13529.729430] Key type .llcrypt registered [13529.895673] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [13545.277984] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [13546.317115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13546.346744] alg: No test for adler32 (adler32-zlib) [13547.482659] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [13547.874351] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [13549.575488] Key type lgssc registered [13551.193487] Lustre: Echo OBD driver; http://www.lustre.org/ [13589.158571] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 19:02:26 (1788476546) [13608.066571] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [13610.669884] Key type lgssc unregistered [13610.954748] LNet: 471659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13610.966289] LNetError: 471659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13610.984978] LNet: Removed LNI 192.168.203.123@tcp [13611.885735] Key type .llcrypt unregistered [13611.888321] Key type ._llcrypt unregistered [13631.734525] Key type ._llcrypt registered [13631.744804] Key type .llcrypt registered [13631.910460] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [13633.175569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13633.342653] alg: No test for adler32 (adler32-zlib) [13634.585340] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [13634.961796] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [13636.751201] Key type lgssc registered [13638.229190] Lustre: Echo OBD driver; http://www.lustre.org/ [13647.861248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13647.896676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13649.153552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13649.205536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13649.315146] Lustre: lustre-MDT0000: new disk, initializing [13649.439388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13649.450260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13653.432064] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [13662.660234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13662.780927] Lustre: 472874:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13662.858935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13662.861447] Lustre: Skipped 1 previous similar message [13662.914419] Lustre: lustre-MDT0001: new disk, initializing [13662.999097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13663.042708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13663.055719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13667.758796] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [13676.980603] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13682.226255] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13688.722257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13688.955846] Lustre: lustre-OST0000: new disk, initializing [13688.959801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13688.964523] Lustre: 474842:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13689.038873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13695.585673] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [13698.674287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13698.681660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13698.752650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13704.432777] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13708.390191] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing lnet_if_list [13713.877569] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13713.919701] LNet: Added LNI 192.168.203.123@tcp10 [8/256/0/180] [13721.468352] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13721.499496] LNet: Added LNI 192.168.203.123@tcp11 [8/256/0/180] [13727.736756] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13727.773818] LNet: Added LNI 192.168.203.123@tcp12 [8/256/0/180] [13732.592346] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13732.619613] LNet: Added LNI 192.168.203.123@tcp13 [8/256/0/180] [13738.004460] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13738.054864] LNet: Added LNI 192.168.203.123@tcp14 [8/256/0/180] [13742.648405] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13742.670891] LNet: Added LNI 192.168.203.123@tcp15 [8/256/0/180] [13747.849383] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13747.896491] LNet: Added LNI 192.168.203.123@tcp16 [8/256/0/180] [13753.145562] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13753.195741] LNet: Added LNI 192.168.203.123@tcp17 [8/256/0/180] [13758.317931] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13758.349593] LNet: Added LNI 192.168.203.123@tcp18 [8/256/0/180] [13763.544990] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13763.599805] LNet: Added LNI 192.168.203.123@tcp19 [8/256/0/180] [13768.209935] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13768.237219] LNet: Added LNI 192.168.203.123@tcp20 [8/256/0/180] [13773.591065] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13773.670939] LNet: Added LNI 192.168.203.123@tcp21 [8/256/0/180] [13780.521463] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13780.571323] LNet: Added LNI 192.168.203.123@tcp22 [8/256/0/180] [13787.930986] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13787.994561] LNet: Added LNI 192.168.203.123@tcp23 [8/256/0/180] [13794.124210] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13794.146367] LNet: Added LNI 192.168.203.123@tcp24 [8/256/0/180] [13800.241128] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13800.284951] LNet: Added LNI 192.168.203.123@tcp25 [8/256/0/180] [13806.441067] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13806.477239] LNet: Added LNI 192.168.203.123@tcp26 [8/256/0/180] [13812.955939] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13813.011237] LNet: Added LNI 192.168.203.123@tcp27 [8/256/0/180] [13820.543098] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13820.593502] LNet: Added LNI 192.168.203.123@tcp28 [8/256/0/180] [13828.543294] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13828.609586] LNet: Added LNI 192.168.203.123@tcp29 [8/256/0/180] [13836.845085] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13836.868335] LNet: Added LNI 192.168.203.123@tcp30 [8/256/0/180] [13844.541549] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13844.577601] LNet: Added LNI 192.168.203.123@tcp31 [8/256/0/180] [13851.560743] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13851.587750] LNet: Added LNI 192.168.203.123@tcp32 [8/256/0/180] [13858.867871] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13858.914432] LNet: Added LNI 192.168.203.123@tcp33 [8/256/0/180] [13866.241893] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13866.281733] LNet: Added LNI 192.168.203.123@tcp34 [8/256/0/180] [13873.005881] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13873.042639] LNet: Added LNI 192.168.203.123@tcp35 [8/256/0/180] [13879.761514] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13879.786432] LNet: Added LNI 192.168.203.123@tcp36 [8/256/0/180] [13886.487519] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13886.508678] LNet: Added LNI 192.168.203.123@tcp37 [8/256/0/180] [13893.608423] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13893.634269] LNet: Added LNI 192.168.203.123@tcp38 [8/256/0/180] [13900.550480] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13900.595114] LNet: Added LNI 192.168.203.123@tcp39 [8/256/0/180] [13908.850445] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13908.890189] LNet: Added LNI 192.168.203.123@tcp40 [8/256/0/180] [13916.967548] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13917.015476] LNet: Added LNI 192.168.203.123@tcp41 [8/256/0/180] [13925.068776] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13925.106868] LNet: Added LNI 192.168.203.123@tcp42 [8/256/0/180] [13933.171665] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13933.211704] LNet: Added LNI 192.168.203.123@tcp43 [8/256/0/180] [13942.481582] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13942.535958] LNet: Added LNI 192.168.203.123@tcp44 [8/256/0/180] [13951.628633] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13951.681979] LNet: Added LNI 192.168.203.123@tcp45 [8/256/0/180] [13961.103668] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13961.143055] LNet: Added LNI 192.168.203.123@tcp46 [8/256/0/180] [13970.552834] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13970.587090] LNet: Added LNI 192.168.203.123@tcp47 [8/256/0/180] [13979.380169] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13979.408626] LNet: Added LNI 192.168.203.123@tcp48 [8/256/0/180] [13988.986400] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13989.040913] LNet: Added LNI 192.168.203.123@tcp49 [8/256/0/180] [13998.223501] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13998.252685] LNet: Added LNI 192.168.203.123@tcp50 [8/256/0/180] [14016.487557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14016.496188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14016.500220] Lustre: Skipped 1 previous similar message [14016.509905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14016.510335] Lustre: Skipped 1 previous similar message [14016.533935] LustreError: Skipped 1 previous similar message [14021.601392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14021.609890] Lustre: Skipped 3 previous similar messages [14022.990041] Lustre: server umount lustre-MDT0000 complete [14026.851201] LustreError: 472864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476986 with bad export cookie 11972004169905346998 [14026.866423] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14026.877556] LustreError: 472864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14027.327591] Lustre: server umount lustre-MDT0001 complete [14045.823798] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [14047.208525] Lustre: 472147:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788476990/real 1788476990] req@00000000a1bbf5df x1875353633202816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788477006 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14047.252084] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14047.268949] Lustre: Skipped 1 previous similar message [14053.347757] Lustre: 472144:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788476996/real 1788476996] req@0000000093fff53a x1875353633202944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788477012 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14056.203437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14058.527170] Lustre: 472144:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788477001/real 1788477001] req@00000000247962cb x1875353633203200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788477017 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14073.311702] LustreError: 474835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14073.328035] LustreError: 474835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14073.401367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14077.411373] LustreError: 474836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14078.589798] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [14081.526778] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14081.549591] LustreError: 474835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14086.627797] LustreError: 474836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14088.262507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14088.805325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14092.920547] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14093.993861] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [14101.751964] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14107.547672] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14129.124400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14129.128168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14129.151943] Lustre: Skipped 1 previous similar message [14134.251184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14134.264165] Lustre: Skipped 2 previous similar messages [14139.365509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14139.374433] Lustre: Skipped 1 previous similar message [14141.407511] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14141.549815] Lustre: server umount lustre-OST0000 complete [14149.608883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14149.635262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14151.880231] Lustre: server umount lustre-MDT0000 complete [14154.720527] LustreError: 501080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14154.740157] LustreError: 501080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14155.794667] LustreError: 472863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477114 with bad export cookie 11972004169905349588 [14155.800104] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14155.824027] LustreError: 472863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14156.436945] Lustre: server umount lustre-MDT0001 complete [14167.642943] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [14170.755468] Key type lgssc unregistered [14171.194174] LNet: 501890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14171.204866] LNetError: 501890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14171.215222] LNet: Removed LNI 192.168.203.123@tcp [14171.221778] LNet: Removed LNI 192.168.203.123@tcp10 [14171.225167] LNet: Removed LNI 192.168.203.123@tcp11 [14171.230243] LNet: Removed LNI 192.168.203.123@tcp12 [14171.239943] LNet: Removed LNI 192.168.203.123@tcp13 [14171.245742] LNet: Removed LNI 192.168.203.123@tcp14 [14171.251257] LNet: Removed LNI 192.168.203.123@tcp15 [14171.253275] LNet: Removed LNI 192.168.203.123@tcp16 [14171.261700] LNet: Removed LNI 192.168.203.123@tcp17 [14171.266788] LNet: Removed LNI 192.168.203.123@tcp18 [14171.271753] LNet: Removed LNI 192.168.203.123@tcp19 [14171.283560] LNet: Removed LNI 192.168.203.123@tcp20 [14171.286456] LNet: Removed LNI 192.168.203.123@tcp21 [14171.290551] LNet: Removed LNI 192.168.203.123@tcp22 [14171.297298] LNet: Removed LNI 192.168.203.123@tcp23 [14171.300665] LNet: Removed LNI 192.168.203.123@tcp24 [14171.303165] LNet: Removed LNI 192.168.203.123@tcp25 [14171.306234] LNet: Removed LNI 192.168.203.123@tcp26 [14171.310740] LNet: Removed LNI 192.168.203.123@tcp27 [14171.313616] LNet: Removed LNI 192.168.203.123@tcp28 [14171.315884] LNet: Removed LNI 192.168.203.123@tcp29 [14171.318935] LNet: Removed LNI 192.168.203.123@tcp30 [14171.321561] LNet: Removed LNI 192.168.203.123@tcp31 [14171.328482] LNet: Removed LNI 192.168.203.123@tcp32 [14171.331322] LNet: Removed LNI 192.168.203.123@tcp33 [14171.334969] LNet: Removed LNI 192.168.203.123@tcp34 [14171.341465] LNet: Removed LNI 192.168.203.123@tcp35 [14171.347543] LNet: Removed LNI 192.168.203.123@tcp36 [14171.351848] LNet: Removed LNI 192.168.203.123@tcp37 [14171.354480] LNet: Removed LNI 192.168.203.123@tcp38 [14171.357448] LNet: Removed LNI 192.168.203.123@tcp39 [14171.373103] LNet: Removed LNI 192.168.203.123@tcp40 [14171.375095] LNet: Removed LNI 192.168.203.123@tcp41 [14171.392700] LNet: Removed LNI 192.168.203.123@tcp42 [14171.395987] LNet: Removed LNI 192.168.203.123@tcp43 [14171.402087] LNet: Removed LNI 192.168.203.123@tcp44 [14171.406114] LNet: Removed LNI 192.168.203.123@tcp45 [14171.412987] LNet: Removed LNI 192.168.203.123@tcp46 [14171.418174] LNet: Removed LNI 192.168.203.123@tcp47 [14171.422567] LNet: Removed LNI 192.168.203.123@tcp48 [14171.424024] LNet: Removed LNI 192.168.203.123@tcp49 [14171.459101] LNet: Removed LNI 192.168.203.123@tcp50 [14172.093457] Key type .llcrypt unregistered [14172.098952] Key type ._llcrypt unregistered [14193.581638] Key type ._llcrypt registered [14193.584996] Key type .llcrypt registered [14193.725409] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [14210.624642] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [14211.924110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14212.014090] alg: No test for adler32 (adler32-zlib) [14213.270806] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [14213.534264] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [14215.215176] Key type lgssc registered [14216.495771] Lustre: Echo OBD driver; http://www.lustre.org/ [14252.425386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14252.462608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14253.668512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14253.687815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14253.754262] Lustre: lustre-MDT0000: new disk, initializing [14253.879269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14253.895781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14257.747072] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [14267.010493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14267.135918] Lustre: 505082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14267.197236] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14267.201672] Lustre: Skipped 1 previous similar message [14267.317189] Lustre: lustre-MDT0001: new disk, initializing [14267.433329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14267.464229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14267.472321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14271.758573] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [14280.717160] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14285.607980] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14292.861675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14293.139408] Lustre: *** cfs_fail_loc=90f, val=0*** [14293.224626] Lustre: lustre-OST0000: new disk, initializing [14293.229187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14293.235226] Lustre: 507101:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14293.318739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14298.786654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14298.797857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14298.868114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14299.314384] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [14307.836667] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14319.584681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14319.591119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14319.600795] LustreError: Skipped 1 previous similar message [14319.605417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14319.620026] Lustre: Skipped 1 previous similar message [14319.637822] Lustre: Skipped 1 previous similar message [14324.717030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14324.729343] Lustre: Skipped 1 previous similar message [14329.312528] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14329.508515] Lustre: server umount lustre-OST0000 complete [14333.920415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14333.936341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14333.958234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14334.953371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14339.847256] Lustre: server umount lustre-MDT0000 complete [14343.486185] LustreError: 505074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477302 with bad export cookie 17340287287345803384 [14343.492292] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14343.499263] LustreError: 505074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14343.912506] Lustre: server umount lustre-MDT0001 complete [14355.272559] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [14358.169846] Key type lgssc unregistered [14358.438354] LNet: 509006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14358.449333] LNetError: 509006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14358.477383] LNet: Removed LNI 192.168.203.123@tcp [14359.550646] Key type .llcrypt unregistered [14359.553369] Key type ._llcrypt unregistered [14380.633950] Key type ._llcrypt registered [14380.635781] Key type .llcrypt registered [14380.797350] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [14381.845633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14381.866454] alg: No test for adler32 (adler32-zlib) [14383.170244] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [14383.619862] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [14385.272197] Key type lgssc registered [14386.733346] Lustre: Echo OBD driver; http://www.lustre.org/ [14395.904230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14395.921852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14397.480993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14402.844362] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [14407.659784] LustreError: 510272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14407.678327] LustreError: 510272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14412.783510] LustreError: 510271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14413.165825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14413.749465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14419.599000] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [14426.931850] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14431.642646] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14437.257560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14437.737887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14443.939361] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [14450.021539] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14461.059600] loop0: detected capacity change from 0 to 204800000 [14461.243498] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14461.251464] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14461.279521] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14467.859326] loop0: detected capacity change from 0 to 204800000 [14468.048437] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14468.060438] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14468.075103] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14473.645951] loop0: detected capacity change from 0 to 204800000 [14479.735105] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14479.841231] Lustre: fs15246-OST0000: new disk, initializing [14479.845499] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14479.852388] Lustre: 513869:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14479.905589] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14485.914909] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [14494.802722] loop1: detected capacity change from 0 to 204800000 [14500.794167] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14500.987665] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14501.061708] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14501.163917] Lustre: fs15246-MDT0000: new disk, initializing [14501.402615] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14501.444721] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14501.575648] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14502.624287] Lustre: 514992:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000121ea925 x1875354417412864/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788477477 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14505.900484] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [14511.718341] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14511.730806] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14511.758256] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14528.485243] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14528.506402] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14532.815339] Lustre: server umount fs15246-MDT0000 complete [14533.620230] LustreError: 510268:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14538.726409] LustreError: 511822:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14541.640510] Lustre: server umount fs15246-OST0000 complete [14554.085930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14554.099845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14554.103663] Lustre: Skipped 1 previous similar message [14554.125114] Lustre: Skipped 1 previous similar message [14559.208767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14559.223611] Lustre: Skipped 1 previous similar message [14564.328506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14564.343028] Lustre: Skipped 1 previous similar message [14566.368225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14566.579880] Lustre: server umount lustre-OST0000 complete [14572.513402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14572.523486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14572.545389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14574.573694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14576.514193] Lustre: server umount lustre-MDT0000 complete [14579.682770] LustreError: 510266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14579.707763] LustreError: 510266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14580.455055] LustreError: 510253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477539 with bad export cookie 16914709681309989362 [14580.466723] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14580.874798] Lustre: server umount lustre-MDT0001 complete [14590.846516] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [14593.648423] Key type lgssc unregistered [14593.881112] LNet: 518088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14593.886922] LNetError: 518088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14593.902284] LNet: Removed LNI 192.168.203.123@tcp [14594.918097] Key type .llcrypt unregistered [14594.922702] Key type ._llcrypt unregistered [14616.516887] Key type ._llcrypt registered [14616.519554] Key type .llcrypt registered [14616.606184] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [14617.691727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14617.859528] alg: No test for adler32 (adler32-zlib) [14618.958406] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [14619.200265] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [14620.871199] Key type lgssc registered [14621.770742] Lustre: Echo OBD driver; http://www.lustre.org/ [14658.977839] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [14669.738732] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [14701.904322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14701.927329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14703.209268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14703.281327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14703.364178] Lustre: lustre-MDT0000: new disk, initializing [14703.449711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14703.460277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14708.323154] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [14718.702322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14718.868465] Lustre: 522552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14718.924984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14718.933906] Lustre: Skipped 1 previous similar message [14719.012353] Lustre: lustre-MDT0001: new disk, initializing [14719.132814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14719.193582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14719.212139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14723.538914] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [14731.444304] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14737.741572] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14746.768346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14747.269826] Lustre: lustre-OST0000: new disk, initializing [14747.273341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14747.279650] Lustre: 524520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14747.378912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14755.273661] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [14756.918648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14756.930239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14757.115532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14764.670212] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14769.790131] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14770.049530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14775.132324] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14775.344807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14777.829702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14777.832341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14777.839078] LustreError: Skipped 1 previous similar message [14777.842416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14777.882089] Lustre: Skipped 1 previous similar message [14782.947512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14782.967832] Lustre: Skipped 2 previous similar messages [14783.602906] Lustre: server umount lustre-OST0000 complete [14788.067194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14788.087779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14788.103242] Lustre: Skipped 1 previous similar message [14793.184706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14793.192822] Lustre: Skipped 2 previous similar messages [14793.641963] Lustre: server umount lustre-MDT0000 complete [14797.560616] LustreError: 522545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477756 with bad export cookie 17099547656334959225 [14797.575904] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14797.578525] LustreError: 522545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14797.958056] Lustre: server umount lustre-MDT0001 complete [14823.547932] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [14835.170235] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [14870.159972] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14549 sec ========= 19:23:47 (1788477827) [14872.283720] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:23:49 (1788477829) === [14876.287119] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:23:53 (1788477833) === [14917.337695] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [14920.148832] Key type lgssc unregistered [14920.388831] LNet: 531166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14920.402209] LNetError: 531166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14920.445721] LNet: Removed LNI 192.168.203.123@tcp [14921.338288] Key type .llcrypt unregistered [14921.343332] Key type ._llcrypt unregistered