[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 455813931 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002356] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007705] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008033] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010020] pid_max: default: 32768 minimum: 301 [ 0.011181] LSM: Security Framework initializing [ 0.013064] Yama: becoming mindful. [ 0.014063] SELinux: Initializing. [ 0.015097] *** VALIDATE selinux *** [ 0.022359] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027313] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028311] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029141] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030169] *** VALIDATE tmpfs *** [ 0.031555] *** VALIDATE proc *** [ 0.032325] *** VALIDATE cgroup *** [ 0.033015] *** VALIDATE cgroup2 *** [ 0.034331] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036023] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039034] Spectre V2 : User space: Vulnerable [ 0.040013] Speculative Store Bypass: Vulnerable [ 0.044750] debug: unmapping init [mem 0xffffffffa8059000-0xffffffffa8060fff] [ 0.047223] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048784] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049035] ... version: 2 [ 0.050018] ... bit width: 48 [ 0.051014] ... generic registers: 4 [ 0.052012] ... value mask: 0000ffffffffffff [ 0.053020] ... max period: 00007fffffffffff [ 0.054022] ... fixed-purpose events: 3 [ 0.055019] ... event mask: 000000070000000f [ 0.056419] rcu: Hierarchical SRCU implementation. [ 0.058874] smp: Bringing up secondary CPUs ... [ 0.059738] x86: Booting SMP configuration: [ 0.060036] .... node #0, CPUs: #1 #2 #3 [ 0.064017] smp: Brought up 1 node, 4 CPUs [ 0.066029] smpboot: Max logical packages: 1 [ 0.067023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.236440] node 0 deferred pages initialised in 167ms [ 0.243232] devtmpfs: initialized [ 0.244314] x86/mm: Memory block size: 128MB [ 0.248404] gcov: version magic: 0x41383552 [ 0.251466] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.256111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.258430] pinctrl core: initialized pinctrl subsystem [ 0.261237] [ 0.261940] ************************************************************* [ 0.264021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.267020] ** ** [ 0.269015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.271020] ** ** [ 0.275020] ** This means that this kernel is built to expose internal ** [ 0.278018] ** IOMMU data structures, which may compromise security on ** [ 0.281021] ** your system. ** [ 0.284019] ** ** [ 0.287017] ** If you see this message and you are not debugging the ** [ 0.289016] ** kernel, report this immediately to your vendor! ** [ 0.292019] ** ** [ 0.294018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.297017] ************************************************************* [ 0.299891] NET: Registered protocol family 16 [ 0.302576] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.305185] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.308083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.313053] cpuidle: using governor menu [ 0.315988] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.318707] PCI: Using configuration type 1 for base access [ 0.321218] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.331143] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.336030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.340082] cryptd: max_cpu_qlen set to 1000 [ 0.343361] ACPI: Added _OSI(Module Device) [ 0.345027] ACPI: Added _OSI(Processor Device) [ 0.348018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.350015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.353910] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.361580] ACPI: Interpreter enabled [ 0.363065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.365020] ACPI: Using IOAPIC for interrupt routing [ 0.367138] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.372467] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.385336] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.388150] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.392026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.396085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.402661] acpiphp: Slot [2] registered [ 0.405207] acpiphp: Slot [5] registered [ 0.406264] acpiphp: Slot [6] registered [ 0.408228] acpiphp: Slot [7] registered [ 0.410183] acpiphp: Slot [8] registered [ 0.412199] acpiphp: Slot [9] registered [ 0.413199] acpiphp: Slot [10] registered [ 0.415253] acpiphp: Slot [3] registered [ 0.417217] acpiphp: Slot [4] registered [ 0.418169] acpiphp: Slot [11] registered [ 0.420175] acpiphp: Slot [12] registered [ 0.422157] acpiphp: Slot [13] registered [ 0.424213] acpiphp: Slot [14] registered [ 0.425217] acpiphp: Slot [15] registered [ 0.427114] acpiphp: Slot [16] registered [ 0.428168] acpiphp: Slot [17] registered [ 0.430160] acpiphp: Slot [18] registered [ 0.432138] acpiphp: Slot [19] registered [ 0.433115] acpiphp: Slot [20] registered [ 0.435156] acpiphp: Slot [21] registered [ 0.437161] acpiphp: Slot [22] registered [ 0.439139] acpiphp: Slot [23] registered [ 0.441142] acpiphp: Slot [24] registered [ 0.442134] acpiphp: Slot [25] registered [ 0.444169] acpiphp: Slot [26] registered [ 0.446144] acpiphp: Slot [27] registered [ 0.448128] acpiphp: Slot [28] registered [ 0.450116] acpiphp: Slot [29] registered [ 0.452173] acpiphp: Slot [30] registered [ 0.453144] acpiphp: Slot [31] registered [ 0.455092] PCI host bridge to bus 0000:00 [ 0.457039] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.460034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.462033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.465034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.468040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.472053] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.474280] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.477204] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.481401] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.492021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.498712] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.501037] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.503024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.506025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.509458] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.513087] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.516061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.519948] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.525022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.541023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.547011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.554302] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.563021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.570022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.588019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.599757] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.608021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.619017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.635024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.648019] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.656021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.665017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.681020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.693085] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.698018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.706020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.732020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.746608] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.756077] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.771022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.793021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.806472] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.815022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.822021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.842019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.852763] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.855470] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.859440] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.862457] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.865303] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.872086] iommu: Default domain type: Passthrough [ 0.873512] SCSI subsystem initialized [ 0.875231] ACPI: bus type USB registered [ 0.877201] usbcore: registered new interface driver usbfs [ 0.880122] usbcore: registered new interface driver hub [ 0.882112] usbcore: registered new device driver usb [ 0.884300] pps_core: LinuxPPS API ver. 1 registered [ 0.887018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.890087] PTP clock support registered [ 0.893048] EDAC MC: Ver: 3.0.0 [ 0.895142] PCI: Using ACPI for IRQ routing [ 0.898077] NetLabel: Initializing [ 0.899014] NetLabel: domain hash size = 128 [ 0.901018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.903130] NetLabel: unlabeled traffic allowed by default [ 0.906220] vgaarb: loaded [ 0.908337] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.910020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.921079] clocksource: Switched to clocksource kvm-clock [ 1.066908] VFS: Disk quotas dquot_6.6.0 [ 1.068753] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.071799] *** VALIDATE ramfs *** [ 1.073337] *** VALIDATE hugetlbfs *** [ 1.074509] pnp: PnP ACPI init [ 1.077269] pnp: PnP ACPI: found 6 devices [ 1.096982] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.100407] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.103021] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.105768] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.108546] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.111559] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.114883] NET: Registered protocol family 2 [ 1.117607] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.122291] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.125483] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.130749] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.134669] TCP: Hash tables configured (established 65536 bind 65536) [ 1.138254] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.141870] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.144678] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.147450] NET: Registered protocol family 1 [ 1.151316] RPC: Registered named UNIX socket transport module. [ 1.153615] RPC: Registered udp transport module. [ 1.157314] RPC: Registered tcp transport module. [ 1.159026] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.161954] NET: Registered protocol family 44 [ 1.163911] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.166201] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.168265] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.170555] PCI: CLS 0 bytes, default 64 [ 1.172378] Unpacking initramfs... [ 2.587615] debug: unmapping init [mem 0xffff93893cc54000-0xffff93893ffbffff] [ 2.594454] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.596712] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.600679] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.114248] Initialise system trusted keyrings [ 3.116067] Key type blacklist registered [ 3.118397] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.127936] zbud: loaded [ 3.130898] *** VALIDATE nfs *** [ 3.132194] *** VALIDATE nfs4 *** [ 3.136340] pstore: using deflate compression [ 3.140104] Platform Keyring initialized [ 3.247175] NET: Registered protocol family 38 [ 3.249067] Key type asymmetric registered [ 3.250800] Asymmetric key parser 'x509' registered [ 3.252914] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.256762] io scheduler mq-deadline registered [ 3.258684] io scheduler kyber registered [ 3.260544] io scheduler bfq registered [ 3.262724] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.265828] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.268284] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.271514] ACPI: Power Button [PWRF] [ 3.277379] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.286538] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.302553] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.321694] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.347208] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.373473] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.404252] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.410212] Non-volatile memory driver v1.3 [ 3.412117] Linux agpgart interface v0.103 [ 3.442951] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.446318] vda: detected capacity change from 0 to 74702848 [ 3.463573] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.467375] vdb: detected capacity change from 0 to 1073741824 [ 3.489101] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.494273] vdc: detected capacity change from 0 to 2621440000 [ 3.520600] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.524738] vdd: detected capacity change from 0 to 2621440000 [ 3.548362] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.552191] vde: detected capacity change from 0 to 4294967296 [ 3.578725] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.582352] vdf: detected capacity change from 0 to 4294967296 [ 3.599820] libphy: Fixed MDIO Bus: probed [ 3.607651] usbcore: registered new interface driver usbserial_generic [ 3.611570] usbserial: USB Serial support registered for generic [ 3.615401] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.619935] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.621631] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.623970] mousedev: PS/2 mouse device common for all mice [ 3.626494] rtc_cmos 00:05: RTC can wake from S4 [ 3.629473] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.630707] rtc_cmos 00:05: registered as rtc0 [ 3.634213] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.637433] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.638051] intel_pstate: CPU model not supported [ 3.644677] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.649509] hid: raw HID events driver (C) Jiri Kosina [ 3.652077] usbcore: registered new interface driver usbhid [ 3.654382] usbhid: USB HID core driver [ 3.655979] drop_monitor: Initializing network drop monitor service [ 3.660814] Initializing XFRM netlink socket [ 3.663174] NET: Registered protocol family 10 [ 3.669059] Segment Routing with IPv6 [ 3.670699] NET: Registered protocol family 17 [ 3.673068] mpls_gso: MPLS GSO support [ 3.679110] RAS: Correctable Errors collector initialized. [ 3.681237] AVX version of gcm_enc/dec engaged. [ 3.683123] AES CTR mode by8 optimization enabled [ 3.770040] sched_clock: Marking stable (3770007427, 0)->(4753799475, -983792048) [ 3.775109] registered taskstats version 1 [ 3.777381] Loading compiled-in X.509 certificates [ 3.779697] zswap: loaded using pool lzo/zbud [ 3.807380] Key type big_key registered [ 3.825750] Key type encrypted registered [ 3.828222] ima: No TPM chip found, activating TPM-bypass! [ 3.831133] ima: Allocated hash algorithm: sha1 [ 3.833387] ima: No architecture policies found [ 3.835294] evm: Initialising EVM extended attributes: [ 3.837447] evm: security.selinux [ 3.838885] evm: security.ima [ 3.840097] evm: security.capability [ 3.841851] evm: HMAC attrs: 0x1 [ 3.847471] rtc_cmos 00:05: setting system clock to 2026-08-13 13:18:04 UTC (1786627084) [ 3.876956] debug: unmapping init [mem 0xffffffffa9003000-0xffffffffa91fffff] [ 3.880601] debug: unmapping init [mem 0xffffffffa7d82000-0xffffffffa8058fff] [ 3.889175] Write protecting the kernel read-only data: 28672k [ 3.892985] debug: unmapping init [mem 0xffffffffa6403000-0xffffffffa65fffff] [ 3.896322] debug: unmapping init [mem 0xffffffffa6d14000-0xffffffffa6dfffff] [ 3.941774] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.951071] systemd[1]: Detected virtualization kvm. [ 3.952932] systemd[1]: Detected architecture x86-64. [ 3.954972] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.984814] systemd[1]: No hostname configured. [ 3.986634] systemd[1]: Set hostname to . [ 3.988814] random: systemd: uninitialized urandom read (16 bytes read) [ 3.991613] systemd[1]: Initializing machine ID from random generator. [ 4.157163] random: systemd: uninitialized urandom read (16 bytes read) [ 4.160903] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.167515] random: systemd: uninitialized urandom read (16 bytes read) [ 4.170330] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.174717] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.860184] device-mapper: uevent: version 1.0.3 [ 4.862645] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ 5.543434] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.746963] virtio_net virtio0 ens2: renamed from eth0 [ 5.842645] scsi host0: ata_piix [ 5.852217] scsi host1: ata_piix [ 5.853760] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.857253] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.232045] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.349600] random: crng init done [ 10.351204] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.894104] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.241237] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.533536] SELinux: Disabled at runtime. [ 12.598948] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.607743] systemd[1]: Detected virtualization kvm. [ 12.610037] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.151992] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.156214] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.162204] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.169255] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.176149] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.184965] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.194872] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 13.319444] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.676595] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.002295] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.059825] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.170763] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.187217] EDAC sbridge: Ver: 1.1.2 [ 15.884905] Key type dns_resolver registered [ 16.184444] NFS: Registering the id_resolver key type [ 16.186170] Key type id_resolver registered [ 16.187431] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg324-server login: [ 45.447921] libcfs: loading out-of-tree module taints kernel. [ 45.497381] Key type ._llcrypt registered [ 45.503806] Key type .llcrypt registered [ 45.641699] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 63.587448] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 64.940716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.954680] alg: No test for adler32 (adler32-zlib) [ 66.243104] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 67.167238] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 68.919245] Key type lgssc registered [ 70.661871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.101276] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 127.942571] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 139.250652] hrtimer: interrupt took 2704079 ns [ 140.267402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 140.315832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 141.714201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 141.747910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 141.841532] Lustre: lustre-MDT0000: new disk, initializing [ 141.904745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 141.917417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 146.807482] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.173612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.244893] Lustre: 6501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 159.269890] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 159.273290] Lustre: Skipped 1 previous similar message [ 159.331050] Lustre: lustre-MDT0001: new disk, initializing [ 159.368362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.384521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 159.390884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 163.880382] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.314406] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 176.594686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 176.816410] Lustre: lustre-OST0000: new disk, initializing [ 176.823901] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 176.832549] Lustre: 8440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 176.881755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.160868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 180.174493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 180.253137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 182.414692] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.436353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.561924] Lustre: lustre-OST0001: new disk, initializing [ 194.565542] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 194.572372] Lustre: 9511:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.631653] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 200.469496] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.328815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 203.344258] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 203.433621] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 210.464649] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 215.778356] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 220.965753] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing check_logdir /tmp/testlogs/ [ 225.693729] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing yml_node [ 229.586968] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 231.808987] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 234.455118] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 236.299825] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 13 09:21:55 EDT 2026 [ 253.131867] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 254.364236] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 269.794265] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.797657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.809569] Lustre: Skipped 3 previous similar messages [ 269.829889] Lustre: Skipped 3 previous similar messages [ 274.391502] Lustre: server umount lustre-MDT0000 complete [ 279.799614] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627360 with bad export cookie 3961020539101944881 [ 279.803779] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 279.839068] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 280.045157] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.046448] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.050959] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 280.089772] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 280.190160] Lustre: Skipped 1 previous similar message [ 284.066068] LustreError: 6510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 284.066491] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 284.109341] Lustre: Skipped 2 previous similar messages [ 289.248264] LustreError: 6509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 289.252031] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 289.264744] LustreError: 6509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 289.289314] Lustre: Skipped 1 previous similar message [ 294.367107] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 294.894669] Lustre: server umount lustre-MDT0001 complete [ 299.484063] Lustre: server umount lustre-OST0000 complete [ 303.570417] Lustre: server umount lustre-OST0001 complete [ 310.510463] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 320.104259] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 354.129923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 354.401632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 354.438797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 354.566720] Lustre: lustre-MDT0000: new disk, initializing [ 354.643782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 354.664246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 359.261156] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 368.393329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 368.574734] Lustre: 15066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 368.644100] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 368.653102] Lustre: Skipped 1 previous similar message [ 368.714429] Lustre: lustre-MDT0001: new disk, initializing [ 368.923208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 369.017771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 369.031898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 373.779957] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 384.137955] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 389.922770] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 395.930529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 396.230862] Lustre: lustre-OST0000: new disk, initializing [ 396.233774] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 396.238246] Lustre: 17034:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 396.316466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 398.031685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 398.045848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 398.158049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 401.357539] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 410.400352] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 414.762283] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 414.989411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 419.461751] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 419.688838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 423.904510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 423.908880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 423.927230] LustreError: Skipped 1 previous similar message [ 423.927877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 423.951445] Lustre: Skipped 1 previous similar message [ 423.960717] Lustre: Skipped 2 previous similar messages [ 427.716424] Lustre: server umount lustre-OST0000 complete [ 434.145897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.166492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 434.177411] Lustre: Skipped 2 previous similar messages [ 437.468668] Lustre: server umount lustre-MDT0000 complete [ 439.265486] LustreError: 15077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 439.293660] LustreError: 15077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 441.115197] LustreError: 15056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627521 with bad export cookie 3961020539101946414 [ 441.117239] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 441.128151] LustreError: 15056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 441.449022] Lustre: server umount lustre-MDT0001 complete [ 445.114273] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:25:24 (1786627524) [ 447.242768] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 448.658419] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:25:27 (1786627527) [ 454.640849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 455.385641] LustreError: 19444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 455.479713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 460.203725] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 465.898547] LustreError: 19445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.933111] LustreError: 19445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 468.493131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.110843] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 482.056539] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 487.976521] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 494.341630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 501.481122] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 511.085917] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 521.696542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 521.701288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.726629] Lustre: Skipped 1 previous similar message [ 521.736962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 524.778913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 527.318817] Lustre: server umount lustre-OST0000 complete [ 535.024241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 535.039689] Lustre: Skipped 1 previous similar message [ 545.251927] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 545.422767] Lustre: server umount lustre-MDT0000 complete [ 548.449895] LustreError: 19424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627629 with bad export cookie 3961020539101947296 [ 548.453324] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 548.458433] LustreError: 19424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 548.808891] Lustre: server umount lustre-MDT0001 complete [ 558.297832] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 561.305957] Key type lgssc unregistered [ 561.540575] LNet: 22981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 561.546735] LNetError: 22981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 561.570150] LNet: Removed LNI 192.168.203.124@tcp [ 562.731311] Key type .llcrypt unregistered [ 562.751561] Key type ._llcrypt unregistered [ 591.487548] Key type ._llcrypt registered [ 591.495811] Key type .llcrypt registered [ 591.601863] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 593.054714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 593.102030] alg: No test for adler32 (adler32-zlib) [ 594.306994] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 594.588392] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 596.319162] Key type lgssc registered [ 597.342372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 604.604633] loop: module loaded [ 604.693355] loop0: detected capacity change from 0 to 204800000 [ 605.000514] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 605.021338] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 605.442246] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 610.775135] loop0: detected capacity change from 0 to 204800000 [ 610.950242] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 610.963486] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 610.979127] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 616.231547] loop0: detected capacity change from 0 to 204800000 [ 621.258543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 621.281184] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 622.660313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 622.716862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 622.781462] Lustre: lustre-MDT0000: new disk, initializing [ 622.875595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 622.909191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 627.571052] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 635.998746] loop1: detected capacity change from 0 to 204800000 [ 641.041710] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 641.427321] Lustre: lustre-OST07c6: new disk, initializing [ 641.430655] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 641.433317] Lustre: Skipped 1 previous similar message [ 641.444259] Lustre: 26167:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 641.503203] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 647.122599] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 648.785664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 648.792557] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 648.827609] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 652.340360] Lustre: 25155:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 652.348404] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 652.354818] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 652.360745] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 652.366529] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 652.374142] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 672.251693] Lustre: 25155:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 672.278593] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 672.293607] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 672.298548] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 672.306518] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 672.314599] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 677.087924] Lustre: 25155:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 677.117354] Lustre: 25155:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 677.133948] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 677.151605] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 677.161594] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 677.176380] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 677.187389] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 677.209814] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 677.222664] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 677.235509] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 677.247354] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 677.265227] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.088374] Lustre: 26850:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 682.108543] Lustre: 26850:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 682.117071] Lustre: 26850:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 682.125891] Lustre: 26850:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.132909] Lustre: 26850:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 682.141764] Lustre: 26850:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.155060] Lustre: 26850:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 682.171939] Lustre: 26850:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.187543] Lustre: 26850:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 682.194267] Lustre: 26850:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.204462] Lustre: 26850:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 682.209367] Lustre: 26850:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.874532] Lustre: 25153:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 689.883471] Lustre: 25153:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 689.889471] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 689.898228] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 689.909491] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 689.921181] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 689.932584] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 689.942201] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 689.950596] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 689.960721] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 689.965913] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 689.971623] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 701.230825] Lustre: 25154:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 701.260075] Lustre: 25154:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 701.274371] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 701.284821] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 701.293481] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 701.302127] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 701.311996] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 701.326327] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 701.338474] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 701.352630] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 701.361027] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 701.372987] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 744.852679] Lustre: server umount lustre-OST07c6 complete [ 763.044877] Lustre: server umount lustre-MDT0000 complete [ 778.131046] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 781.094209] Key type lgssc unregistered [ 781.427984] LNet: 28728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.438645] LNetError: 28728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.453485] LNet: Removed LNI 192.168.203.124@tcp [ 782.403926] Key type .llcrypt unregistered [ 782.406549] Key type ._llcrypt unregistered [ 798.936268] Key type ._llcrypt registered [ 798.940603] Key type .llcrypt registered [ 799.068439] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 799.953209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 799.961684] alg: No test for adler32 (adler32-zlib) [ 800.910223] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 801.126823] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 802.792859] Key type lgssc registered [ 803.941865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 811.555212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 811.568965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 813.113847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.838847] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 823.265762] LustreError: 30000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 823.283818] LustreError: 30000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 827.948753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 828.339048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 832.899285] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 840.433375] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 846.072153] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 852.597743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 853.052547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.151039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 859.401571] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 866.332530] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 879.590744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 879.592454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 879.594244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 879.606196] Lustre: Skipped 1 previous similar message [ 879.631194] Lustre: Skipped 1 previous similar message [ 884.716721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 884.723845] Lustre: Skipped 1 previous similar message [ 885.532813] Lustre: server umount lustre-OST0000 complete [ 889.824514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 889.827401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.843314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 889.865566] Lustre: Skipped 1 previous similar message [ 894.946666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 894.952864] Lustre: Skipped 2 previous similar messages [ 895.457659] Lustre: server umount lustre-MDT0000 complete [ 898.821160] LustreError: 29980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627979 with bad export cookie 9941180017433818082 [ 898.828206] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 898.830484] LustreError: 29980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 899.200313] Lustre: server umount lustre-MDT0001 complete [ 910.377381] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 913.265568] Key type lgssc unregistered [ 913.547765] LNet: 33738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.569803] LNetError: 33738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.594078] LNet: Removed LNI 192.168.203.124@tcp [ 914.596282] Key type .llcrypt unregistered [ 914.599299] Key type ._llcrypt unregistered [ 932.529364] Key type ._llcrypt registered [ 932.533898] Key type .llcrypt registered [ 932.642657] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 933.560086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 933.610892] alg: No test for adler32 (adler32-zlib) [ 934.714910] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 935.033604] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 936.760808] Key type lgssc registered [ 938.122252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 946.240366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 946.259483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.908691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 952.849991] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 957.411902] LustreError: 35009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 957.437853] LustreError: 35009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 961.537788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.935603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 966.435786] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 973.361845] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 978.358149] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 983.854790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 984.061949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 989.492901] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 993.283099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 999.604804] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1012.704800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1012.717715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.732068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1013.731569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1013.735389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1013.751690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.783367] Lustre: Skipped 1 previous similar message [ 1016.808926] Lustre: server umount lustre-OST0000 complete [ 1023.459218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1023.469102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.482776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.663800] Lustre: server umount lustre-MDT0000 complete [ 1029.089440] LustreError: 35005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1029.123158] LustreError: 35005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1029.968719] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628110 with bad export cookie 10638969360603059918 [ 1029.979256] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1029.979730] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1030.290320] Lustre: server umount lustre-MDT0001 complete [ 1040.027947] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1042.610443] Key type lgssc unregistered [ 1042.841905] LNet: 38545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.853242] LNetError: 38545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.873712] LNet: Removed LNI 192.168.203.124@tcp [ 1043.604576] Key type .llcrypt unregistered [ 1043.608300] Key type ._llcrypt unregistered [ 1068.496624] Key type ._llcrypt registered [ 1068.506598] Key type .llcrypt registered [ 1068.664656] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1069.837398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1069.901040] alg: No test for adler32 (adler32-zlib) [ 1070.999648] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1071.298708] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1072.959247] Key type lgssc registered [ 1074.167623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1081.710303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1081.753128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1083.209030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.344385] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1092.583724] LustreError: 39964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1092.598797] LustreError: 39964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1095.860023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.228074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1100.254679] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1107.326952] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1112.096775] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1117.774268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1118.106639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1123.747964] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1128.379275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1130.081220] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1137.012739] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 1137.121883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1137.137613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.523238] Lustre: server umount lustre-MDT0000 complete [ 1138.658492] LustreError: 41000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.665539] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.433681] LustreError: 39944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628221 with bad export cookie 7798462660259354240 [ 1140.435699] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1140.442759] LustreError: 39944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1142.114590] LustreError: 41203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.135613] LustreError: 41203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1142.155862] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.24@tcp (stopping) [ 1142.162079] Lustre: Skipped 1 previous similar message [ 1143.776844] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1143.789757] Lustre: Skipped 1 previous similar message [ 1143.798045] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1145.824295] LustreError: 39960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1145.827766] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1145.842274] LustreError: 39960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1146.847830] Lustre: server umount lustre-MDT0001 complete [ 1167.247365] Lustre: server umount lustre-OST0000 complete [ 1182.721061] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1185.413264] Key type lgssc unregistered [ 1185.759647] LNet: 43824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1185.767069] LNetError: 43824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1185.782737] LNet: Removed LNI 192.168.203.124@tcp [ 1186.551310] Key type .llcrypt unregistered [ 1186.555087] Key type ._llcrypt unregistered [ 1205.156059] Key type ._llcrypt registered [ 1205.168247] Key type .llcrypt registered [ 1205.377141] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1206.101816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1206.113137] alg: No test for adler32 (adler32-zlib) [ 1207.101395] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1207.258933] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1208.887489] Key type lgssc registered [ 1210.127366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1221.182817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1221.205477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.966316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.222405] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1232.361946] LustreError: 45222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.383360] LustreError: 45222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1237.473814] LustreError: 45223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1239.579675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1240.329283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1245.118554] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1252.841766] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1257.711816] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1263.818985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1264.176353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1268.270646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1269.679363] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1276.686975] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1282.929831] Lustre: lustre-OST0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 1283.556916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.562675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1283.563587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1283.581829] Lustre: Skipped 1 previous similar message [ 1287.650682] Lustre: server umount lustre-OST0000 complete [ 1307.104516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1307.114673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.133740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1307.138357] Lustre: Skipped 1 previous similar message [ 1309.153865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.156437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1309.183670] Lustre: Skipped 1 previous similar message [ 1310.652382] Lustre: server umount lustre-MDT0000 complete [ 1313.929252] LustreError: 45203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628394 with bad export cookie 14305212130522770790 [ 1313.943709] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1313.948728] LustreError: 45203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1320.348826] Lustre: server umount lustre-MDT0001 complete [ 1330.054377] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1332.787837] Key type lgssc unregistered [ 1333.142101] LNet: 48938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1333.157715] LNetError: 48938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1333.181271] LNet: Removed LNI 192.168.203.124@tcp [ 1334.270749] Key type .llcrypt unregistered [ 1334.275392] Key type ._llcrypt unregistered [ 1358.077094] Key type ._llcrypt registered [ 1358.082252] Key type .llcrypt registered [ 1358.199950] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1359.313599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1359.414780] alg: No test for adler32 (adler32-zlib) [ 1360.634868] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1360.902731] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1362.591238] Key type lgssc registered [ 1363.934452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1370.902536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1370.931347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.445321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1376.882980] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1382.883484] LustreError: 50352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1382.904405] LustreError: 50352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1383.862556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1384.146913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1388.166494] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1393.045345] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1397.783183] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1403.307378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1403.709671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1408.802637] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1416.246258] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1418.034544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1426.472823] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1429.770427] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1441.755310] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-13 9h42m00s [ 1442.887341] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1445.863620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1445.873252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.889162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1446.951769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1446.957695] Lustre: Skipped 4 previous similar messages [ 1448.807190] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 1450.614745] Lustre: server umount lustre-MDT0000 complete [ 1453.027456] LustreError: 50348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1453.040179] LustreError: 50348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1453.588050] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1459.123708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1459.170883] LustreError: 51080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1459.187389] LustreError: 51080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1459.258548] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1459.447308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1459.496168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1463.723401] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1465.901773] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1467.373583] Lustre: DEBUG MARKER: Wait for df (36723) ... [ 1468.908413] Lustre: DEBUG MARKER: done [ 1475.044170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1475.063056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.092507] Lustre: Skipped 2 previous similar messages [ 1475.113993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1475.116868] Lustre: Skipped 1 previous similar message [ 1481.195892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1481.207397] Lustre: Skipped 3 previous similar messages [ 1486.303334] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1486.378515] Lustre: server umount lustre-OST0000 complete [ 1491.429187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1491.446766] Lustre: Skipped 1 previous similar message [ 1491.458881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1491.462263] Lustre: Skipped 3 previous similar messages [ 1496.906350] Lustre: server umount lustre-MDT0000 complete [ 1497.569320] LustreError: 51825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1497.588582] LustreError: 51825:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1501.018055] LustreError: 52980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628581 with bad export cookie 7752152508850429663 [ 1501.018558] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1501.029149] LustreError: 52980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1501.333543] Lustre: server umount lustre-MDT0001 complete [ 1511.079959] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1513.784590] Key type lgssc unregistered [ 1514.080681] LNet: 55279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1514.085906] LNetError: 55279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1514.104952] LNet: Removed LNI 192.168.203.124@tcp [ 1514.844534] Key type .llcrypt unregistered [ 1514.849750] Key type ._llcrypt unregistered [ 1550.428381] Key type ._llcrypt registered [ 1550.429936] Key type .llcrypt registered [ 1550.511632] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1551.688296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1551.723243] alg: No test for adler32 (adler32-zlib) [ 1552.745464] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1552.961194] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1554.631843] Key type lgssc registered [ 1555.617754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1562.662559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1562.678592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.035930] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1564.074923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1564.259119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1568.054773] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1575.549045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1575.661758] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1575.687118] Lustre: 57488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1575.914747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1579.928827] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1586.282309] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1591.080812] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1597.021673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1597.230826] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1597.422175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1602.231439] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1606.277969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1607.837796] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1615.157944] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-13 9h44m53s [ 1616.502239] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1623.115327] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1625.482645] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1625.589593] LustreError: 57493:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1645.989172] Lustre: lustre-MDT0000: Client c5fe89f5-0e84-4f08-b454-dd9ac2d56afb (at 192.168.203.24@tcp) reconnecting [ 1665.663136] LustreError: 57493:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1667.694083] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1669.639957] Lustre: DEBUG MARKER: done [ 1677.795770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1677.801208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1677.813797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1678.311514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.328534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1678.336885] Lustre: Skipped 1 previous similar message [ 1680.405975] Lustre: server umount lustre-OST0000 complete [ 1688.545152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.546384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1688.553803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1688.563152] Lustre: Skipped 1 previous similar message [ 1689.751842] Lustre: server umount lustre-MDT0000 complete [ 1692.901494] LustreError: 57479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628773 with bad export cookie 589882058649939174 [ 1692.910475] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1692.911343] LustreError: 57479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1693.180795] Lustre: server umount lustre-MDT0001 complete [ 1701.703923] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1704.112917] Key type lgssc unregistered [ 1704.355882] LNet: 61816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.364276] LNetError: 61816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.375381] LNet: Removed LNI 192.168.203.124@tcp [ 1705.162243] Key type .llcrypt unregistered [ 1705.166175] Key type ._llcrypt unregistered [ 1739.949571] Key type ._llcrypt registered [ 1739.952663] Key type .llcrypt registered [ 1740.062891] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1741.104783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1741.127276] alg: No test for adler32 (adler32-zlib) [ 1742.194693] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1742.484471] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1744.167363] Key type lgssc registered [ 1745.117839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1747.903043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1747.927554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1749.190438] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1749.230704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1749.356202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.331847] Lustre: Failing over lustre-MDT0000 [ 1750.715257] Lustre: server umount lustre-MDT0000 complete [ 1756.208144] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:47:15 (1786628835) [ 1761.934805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1762.267744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1766.677305] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1773.761105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1773.841058] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1773.872524] Lustre: 64592:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1774.097919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1777.768461] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1783.959508] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1788.983680] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1794.845478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1795.004371] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1795.177477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.199464] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1802.924783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1807.992911] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1814.200116] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1823.750386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.753104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1823.755786] Lustre: Skipped 2 previous similar messages [ 1823.770912] Lustre: Skipped 2 previous similar messages [ 1826.633260] Lustre: server umount lustre-MDT0000 complete [ 1830.070687] LustreError: 64585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628910 with bad export cookie 7220168024295420880 [ 1830.079924] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1830.091136] LustreError: 64585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1830.284260] Lustre: server umount lustre-MDT0001 complete [ 1833.360501] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1837.386929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1845.780886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1850.336301] Lustre: 63518:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786628914/real 1786628914] req@00000000a1b3f86d x1873416104746624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786628930 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1850.369172] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1851.362720] Lustre: 63519:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786628916/real 1786628916] req@0000000033bba030 x1873416104746752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786628932 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1854.431156] Lustre: 63517:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786628919/real 1786628919] req@000000004ccdcec0 x1873416104746880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786628935 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1854.498598] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6433309062c33cb3 [ 1854.512935] Lustre: MGC192.168.203.124@tcp: Connection restored to 0@lo (at 0@lo) [ 1854.744601] LustreError: 66439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1854.768853] LustreError: 66439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1854.833448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.925656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1855.842429] LustreError: 66440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1858.854982] LustreError: 66439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1858.984051] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1860.898828] LustreError: 68362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1860.917906] LustreError: 68362:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1861.919314] Lustre: 63518:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786628925/real 1786628925] req@00000000f7f2383d x1873416104747264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786628941 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1861.970458] Lustre: 63518:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1866.016519] Lustre: 63519:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786628930/real 1786628930] req@00000000d5faf41b x1873416104747392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786628946 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1866.219120] LustreError: 68362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1866.234287] LustreError: 68362:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1866.757942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1867.079911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1870.754030] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1872.356438] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1876.718311] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1878.501330] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1882.057765] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1887.713980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1887.724193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1887.745694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1888.747159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1888.752732] Lustre: Skipped 2 previous similar messages [ 1889.686909] Lustre: server umount lustre-MDT0000 complete [ 1892.939335] LustreError: 67502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628973 with bad export cookie 7220168024295423155 [ 1892.942057] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1892.949842] LustreError: 67502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1893.159657] Lustre: server umount lustre-MDT0001 complete [ 1898.974640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1906.493464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.239447] Lustre: 63517:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786628974/real 1786628974] req@000000008eb773c2 x1873416104772096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786628990 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1910.267794] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1910.282067] Lustre: Skipped 2 previous similar messages [ 1917.471822] LustreError: 63515:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ad2d6372 x1873416104773248/t0(0) o250->MGC192.168.203.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1917.722403] LustreError: 66439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1917.789975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1917.851835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1922.610883] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1929.189529] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1930.222212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.067808] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1935.858408] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1941.731473] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1946.423135] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1953.830697] Lustre: server umount lustre-MDT0000 complete [ 1956.320055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1956.326898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.329255] LustreError: 71781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1956.329263] LustreError: 71781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 1956.363438] Lustre: Skipped 2 previous similar messages [ 1956.522684] LustreError: 64585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629037 with bad export cookie 7220168024295424534 [ 1956.523750] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1956.534554] LustreError: 64585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1956.735441] Lustre: server umount lustre-MDT0001 complete [ 1962.317508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1965.428940] Lustre: DEBUG MARKER: files compared the same [ 1977.043601] Lustre: server umount lustre-OST0000 complete [ 1991.862236] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1994.336756] Key type lgssc unregistered [ 1994.574830] LNet: 74326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1994.583241] LNetError: 74326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1994.598232] LNet: Removed LNI 192.168.203.124@tcp [ 1995.223715] Key type .llcrypt unregistered [ 1995.227822] Key type ._llcrypt unregistered [ 2018.208174] Key type ._llcrypt registered [ 2018.209616] Key type .llcrypt registered [ 2018.293967] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2019.495069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2019.584596] alg: No test for adler32 (adler32-zlib) [ 2021.044926] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2021.374428] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2023.103722] Key type lgssc registered [ 2024.138906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2030.305956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2030.316937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.564748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2035.487776] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug +malloc all [ 2041.824767] LustreError: 75740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.856191] LustreError: 75740:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2043.355045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2043.663668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2047.352057] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug +malloc all [ 2054.451187] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2059.560409] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2064.864092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2065.168962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.844396] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug +malloc all [ 2076.460888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2077.420429] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2086.890062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2086.891152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.905669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2086.907748] Lustre: Skipped 1 previous similar message [ 2091.210466] Lustre: server umount lustre-OST0000 complete [ 2097.121823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.128631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.142809] Lustre: Skipped 1 previous similar message [ 2097.145953] Lustre: Skipped 1 previous similar message [ 2101.473345] Lustre: server umount lustre-MDT0000 complete [ 2102.241158] LustreError: 75737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2102.270997] LustreError: 75737:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2104.956422] LustreError: 75722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629185 with bad export cookie 8361543622569497154 [ 2104.966413] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2104.967357] LustreError: 75722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2105.291791] Lustre: server umount lustre-MDT0001 complete [ 2117.190370] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2119.720835] Key type lgssc unregistered [ 2119.982616] LNet: 79274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2119.992149] LNetError: 79274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2121.066222] LNet: Removed LNI 192.168.203.124@tcp [ 2121.811719] Key type .llcrypt unregistered [ 2121.816778] Key type ._llcrypt unregistered [ 2146.664411] Key type ._llcrypt registered [ 2146.670860] Key type .llcrypt registered [ 2146.812645] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2149.279680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2149.319769] alg: No test for adler32 (adler32-zlib) [ 2150.543100] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2150.768898] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2152.423198] Key type lgssc registered [ 2153.786057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2161.204571] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2161.222331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2176.735991] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2181.855341] LustreError: 80665:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.124@tcp: failed processing log, type 4: rc = -110 [ 2208.479228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2215.374407] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2221.825313] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2228.757675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.914561] Lustre: *** cfs_fail_loc=706, val=0*** [ 2228.932513] LustreError: 80001:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000021263e40 x1873416532527360/t0(0) o250->MGC192.168.203.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2229.299605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.398238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2233.590257] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2234.346936] LustreError: 80691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2234.370470] LustreError: 80691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2239.474766] LustreError: 80690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2241.277655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.585080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2246.381484] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2253.929623] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2259.507602] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2261.985574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2262.001793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2262.021731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2263.010808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2263.014572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2263.042255] Lustre: Skipped 1 previous similar message [ 2267.674524] Lustre: server umount lustre-OST0000 complete [ 2272.225206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2272.242864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.266941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2277.213464] Lustre: server umount lustre-MDT0000 complete [ 2278.372900] LustreError: 82536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.389943] LustreError: 82536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2280.695728] LustreError: 80672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629361 with bad export cookie 4955357204739625218 [ 2280.697385] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2280.707543] LustreError: 80672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2281.002414] Lustre: server umount lustre-MDT0001 complete [ 2291.476029] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2293.971806] Key type lgssc unregistered [ 2294.250704] LNet: 84283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.264692] LNetError: 84283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2294.283356] LNet: Removed LNI 192.168.203.124@tcp [ 2295.277444] Key type .llcrypt unregistered [ 2295.280581] Key type ._llcrypt unregistered [ 2318.662567] Key type ._llcrypt registered [ 2318.665247] Key type .llcrypt registered [ 2318.826657] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2319.971745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2320.164247] alg: No test for adler32 (adler32-zlib) [ 2321.187168] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2321.400439] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2323.039191] Key type lgssc registered [ 2324.473075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2333.268846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2333.321452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2339.336798] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2346.651706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2347.149161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2351.800940] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2352.607874] LustreError: 86388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.640848] LustreError: 86388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2356.713962] LustreError: 86392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.739084] LustreError: 86392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2360.007273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2360.401919] LustreError: 86388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.422538] LustreError: 86388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2360.444191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2365.922056] LustreError: 86391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.947163] LustreError: 86391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2366.490517] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2371.040882] LustreError: 86392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.065551] LustreError: 86392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2374.150309] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2374.159963] Lustre: MGS: Client f119ef87-a1f3-4177-9a79-952062fc978e (at 0@lo) reconnecting [ 2374.191879] LustreError: 86402:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2374.481438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2374.558125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2378.828578] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2395.110436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2395.116573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2395.125258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2397.257485] Lustre: server umount lustre-OST0000 complete [ 2405.349898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2405.357625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.364820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2405.380132] Lustre: Skipped 2 previous similar messages [ 2405.383095] Lustre: Skipped 1 previous similar message [ 2410.470536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2410.478191] Lustre: Skipped 2 previous similar messages [ 2414.562514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2415.071157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2415.178406] Lustre: server umount lustre-MDT0000 complete [ 2418.383516] Lustre: server umount lustre-MDT0001 complete [ 2421.542796] Lustre: server umount MGS complete [ 2430.483460] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:58:29 (1786629509) [ 2456.819631] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2466.135985] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2497.314885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2501.009157] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2511.651432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2511.810772] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2511.862473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2511.913128] Lustre: lustre-MDT0001: new disk, initializing [ 2511.995619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2512.015097] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2512.027480] Lustre: 93554:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2512.040370] LustreError: 93554:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2515.801729] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2525.809707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2526.000906] Lustre: lustre-OST0000: new disk, initializing [ 2526.008317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2526.013415] Lustre: 94321:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2526.060223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2532.115738] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2534.496764] Lustre: 94755:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000006f73c3f7 x1873416711872640/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786629631 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2540.167305] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2545.919833] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2545.934956] Lustre: MGS: Client b75a3fac-5c29-4d87-95e6-4fe92061373d (at 0@lo) reconnecting [ 2545.968964] Lustre: 92824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2545.980734] LustreError: 93540:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2545.992252] LustreError: 93540:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2546.019041] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2546.105974] Lustre: lustre-MDT0000: new disk, initializing [ 2546.210810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.236878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2546.349956] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2547.296120] Lustre: 95392:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000023507bcc x1873416711879040/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786629643 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2549.781052] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2552.361219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2552.374525] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2552.450245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2573.282393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2573.300420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.314974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2573.319862] Lustre: Skipped 1 previous similar message [ 2575.328452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2575.338508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2579.512774] Lustre: server umount lustre-OST0000 complete [ 2587.617585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2587.648326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.662188] Lustre: Skipped 4 previous similar messages [ 2597.343166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2597.463239] Lustre: server umount lustre-MDT0000 complete [ 2597.869873] LustreError: 93526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.881488] LustreError: 93526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2615.265860] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2615.485145] Lustre: server umount lustre-MDT0001 complete [ 2619.379992] Lustre: server umount MGS complete [ 2629.570452] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:01:48 (1786629708) [ 2646.399258] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2648.637642] Key type lgssc unregistered [ 2648.833051] LNet: 97906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2648.840548] LNetError: 97906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2648.851588] LNet: Removed LNI 192.168.203.124@tcp [ 2649.509910] Key type .llcrypt unregistered [ 2649.513286] Key type ._llcrypt unregistered [ 2664.202824] Key type ._llcrypt registered [ 2664.204746] Key type .llcrypt registered [ 2664.318378] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2665.274779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2665.360091] alg: No test for adler32 (adler32-zlib) [ 2666.438566] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2666.653836] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2668.359511] Key type lgssc registered [ 2669.286863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2676.626935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2676.645482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.131942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.446331] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2687.471675] LustreError: 99109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.491583] LustreError: 99109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2690.695629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.972832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2695.277966] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2702.124831] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2707.534456] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2714.116028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2714.337041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2720.903824] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2722.552687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2727.530743] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2736.096335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2736.100242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2736.119751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2738.155798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2738.156421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2738.163369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.600858] Lustre: server umount lustre-OST0000 complete [ 2747.366056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2747.377227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.394100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2747.399335] Lustre: Skipped 1 previous similar message [ 2750.812859] Lustre: server umount lustre-MDT0000 complete [ 2753.505188] LustreError: 99109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.515292] LustreError: 99109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2753.630367] LustreError: 99091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629834 with bad export cookie 4676028045950536975 [ 2753.633450] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2753.636242] LustreError: 99091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2753.872943] Lustre: server umount lustre-MDT0001 complete [ 2763.201973] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2766.107325] Key type lgssc unregistered [ 2766.435544] LNet: 102643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2766.447378] LNetError: 102643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2766.460883] LNet: Removed LNI 192.168.203.124@tcp [ 2767.254104] Key type .llcrypt unregistered [ 2767.264499] Key type ._llcrypt unregistered [ 2780.743961] Key type ._llcrypt registered [ 2780.745632] Key type .llcrypt registered [ 2780.808116] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2781.707942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2781.737341] alg: No test for adler32 (adler32-zlib) [ 2782.771130] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2782.987939] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2784.663241] Key type lgssc registered [ 2785.830791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2792.571933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2792.572670] LustreError: 103707:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2792.579851] Lustre: Skipped 1 previous similar message [ 2792.579870] LustreError: 103706:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2792.597905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2792.605783] LustreError: 103707:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 2792.605811] LustreError: 103707:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2792.605829] LustreError: 103707:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2794.178413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.154202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2801.361590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2805.019464] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2808.948445] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing lsmod [ 2811.086936] LustreError: 104909:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2811.122387] LustreError: 104915:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2811.126545] LustreError: 104909:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4964 [ 2811.131505] LustreError: 104909:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2811.137971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2811.140033] LustreError: 104909:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2811.433475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2813.736973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2813.744086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2813.762163] Lustre: Skipped 1 previous similar message [ 2813.782416] Lustre: Skipped 2 previous similar messages [ 2819.066831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2819.079658] Lustre: Skipped 2 previous similar messages [ 2824.162188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2824.166660] Lustre: Skipped 1 previous similar message [ 2826.210360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2826.218235] Lustre: Skipped 2 previous similar messages [ 2828.256361] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2830.405149] Lustre: server umount lustre-MDT0000 complete [ 2831.329033] LustreError: 103744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.354114] LustreError: 103744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2834.104416] LustreError: 103725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629914 with bad export cookie 2989207442020196148 [ 2834.117530] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2834.129930] LustreError: 103725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2834.359257] LustreError: 104175:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000093b840b7 x1873417195246848/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2834.471669] Lustre: server umount lustre-MDT0001 complete [ 2848.494609] Lustre: server umount lustre-OST0000 complete [ 2855.803102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2856.194163] LustreError: 105982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2856.289713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2860.199092] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2861.536138] LustreError: 105983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2865.638724] LustreError: 105982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.471669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.756915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2872.085626] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2879.303817] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2883.871745] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2889.864061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2890.201502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.061485] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2900.466934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2902.325440] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2908.646379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2908.656695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2908.684775] Lustre: Skipped 1 previous similar message [ 2908.689903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2910.694165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.654809] Lustre: server umount lustre-OST0000 complete [ 2918.880733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2918.892371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.929155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2918.938320] Lustre: Skipped 2 previous similar messages [ 2931.167371] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2931.370448] Lustre: server umount lustre-MDT0000 complete [ 2934.731504] LustreError: 105963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630015 with bad export cookie 2989207442020196848 [ 2934.737930] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2934.745187] LustreError: 105963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2935.078870] Lustre: server umount lustre-MDT0001 complete [ 2944.392724] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2947.502254] Key type lgssc unregistered [ 2947.841780] LNet: 109517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2947.852767] LNetError: 109517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2948.904228] LNet: Removed LNI 192.168.203.124@tcp [ 2949.861308] Key type .llcrypt unregistered [ 2949.866077] Key type ._llcrypt unregistered [ 2973.828578] Key type ._llcrypt registered [ 2973.832190] Key type .llcrypt registered [ 2973.963857] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2974.762422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2974.788779] alg: No test for adler32 (adler32-zlib) [ 2975.843905] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2976.069516] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2977.735187] Key type lgssc registered [ 2978.762570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2985.307161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2985.325542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2986.736345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2992.027470] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2997.219084] LustreError: 110939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.237031] LustreError: 110939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3000.341437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3000.673690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3005.689941] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3012.820574] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3018.432635] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3024.890585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.398924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3031.549903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3032.187470] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3039.413752] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3045.268630] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3048.687691] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3052.002443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3052.010836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3052.012609] LustreError: Skipped 1 previous similar message [ 3052.014442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3052.052211] Lustre: Skipped 1 previous similar message [ 3054.064546] Lustre: server umount lustre-OST0000 complete [ 3062.241026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3062.243023] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3062.251664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3062.251672] Lustre: Skipped 1 previous similar message [ 3062.287753] Lustre: Skipped 1 previous similar message [ 3063.623940] Lustre: server umount lustre-MDT0000 complete [ 3067.303951] LustreError: 110918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630147 with bad export cookie 16980193907608292345 [ 3067.307800] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3067.317577] LustreError: 110918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3067.713592] Lustre: server umount lustre-MDT0001 complete [ 3077.621350] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 3080.375283] Key type lgssc unregistered [ 3080.681168] LNet: 114578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3080.688457] LNetError: 114578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3080.706136] LNet: Removed LNI 192.168.203.124@tcp [ 3081.572953] Key type .llcrypt unregistered [ 3081.575363] Key type ._llcrypt unregistered [ 3098.147214] Key type ._llcrypt registered [ 3098.151160] Key type .llcrypt registered [ 3098.269291] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3099.338985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3099.357161] alg: No test for adler32 (adler32-zlib) [ 3100.426939] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 3100.642931] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 3102.327396] Key type lgssc registered [ 3103.419693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3110.559087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3110.574672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.010687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.320590] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3122.150252] LustreError: 115804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3122.178496] LustreError: 115804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3126.616753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3127.105948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3131.425711] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3137.982347] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3145.013816] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3154.507971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3155.071356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3157.189808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3163.809396] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3170.697247] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3182.048511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3182.055656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3182.063507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3182.069580] Lustre: Skipped 1 previous similar message [ 3185.089807] Lustre: server umount lustre-OST0000 complete [ 3188.705468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3188.712054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.731336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3188.739171] Lustre: Skipped 1 previous similar message [ 3192.288769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3192.291469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3192.317205] Lustre: Skipped 1 previous similar message [ 3194.359182] Lustre: server umount lustre-MDT0000 complete [ 3197.153315] LustreError: 117647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630277 with bad export cookie 17234808196170446608 [ 3197.158707] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3197.160731] LustreError: 117647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3197.405517] Lustre: server umount lustre-MDT0001 complete [ 3205.712660] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 3208.045708] Key type lgssc unregistered [ 3208.281139] LNet: 119342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3208.287891] LNetError: 119342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3208.303664] LNet: Removed LNI 192.168.203.124@tcp [ 3209.023329] Key type .llcrypt unregistered [ 3209.026703] Key type ._llcrypt unregistered [ 3230.561508] Key type ._llcrypt registered [ 3230.563584] Key type .llcrypt registered [ 3230.730801] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3232.044489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3232.109656] alg: No test for adler32 (adler32-zlib) [ 3233.124422] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 3233.346973] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 3235.055182] Key type lgssc registered [ 3236.065648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3242.201588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3242.212043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3243.583264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.623884] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3253.736750] LustreError: 120753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3253.757952] LustreError: 120753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3255.495083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.048890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3261.461236] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3269.814268] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3275.823252] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3281.909814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3282.260086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3287.087567] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3288.432743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3293.597786] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3308.038801] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3311.063822] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3311.068874] Lustre: Skipped 2 previous similar messages [ 3314.838325] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3314.846728] Lustre: Skipped 2 previous similar messages [ 3318.350636] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3318.395879] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3323.347744] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.24@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3323.355578] Lustre: Skipped 2 previous similar messages [ 3325.195411] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.24@tcp 192.168.0.[2,10]@tcp [ 3330.025832] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.24/24@tcp in log lustre-MDT0000 [ 3330.033244] Lustre: Skipped 2 previous similar messages [ 3330.945658] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.24/24@tcp [ 3330.950219] Lustre: Skipped 1 previous similar message [ 3336.170568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3336.173799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3336.185831] LustreError: Skipped 1 previous similar message [ 3336.204537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3336.676173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3336.682247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3341.437128] Lustre: server umount lustre-OST0000 complete [ 3346.920440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3346.947279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3346.962438] Lustre: Skipped 2 previous similar messages [ 3352.036817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3352.040689] Lustre: Skipped 1 previous similar message [ 3357.156156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3357.169918] Lustre: Skipped 1 previous similar message [ 3359.201490] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3359.436113] Lustre: server umount lustre-MDT0000 complete [ 3362.274385] LustreError: 120750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.293452] LustreError: 120750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3362.701707] LustreError: 124101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630443 with bad export cookie 14350296924636133176 [ 3362.705891] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3362.717771] LustreError: 124101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3363.089401] Lustre: server umount lustre-MDT0001 complete [ 3372.869966] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 3375.408475] Key type lgssc unregistered [ 3375.663562] LNet: 125470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3375.673797] LNetError: 125470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3375.688204] LNet: Removed LNI 192.168.203.124@tcp [ 3376.472440] Key type .llcrypt unregistered [ 3376.475198] Key type ._llcrypt unregistered [ 3400.055939] Key type ._llcrypt registered [ 3400.058282] Key type .llcrypt registered [ 3400.147155] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3400.877957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3400.964071] alg: No test for adler32 (adler32-zlib) [ 3402.072967] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 3402.268562] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 3403.983182] Key type lgssc registered [ 3405.003738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3411.279379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3411.305032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3412.725928] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3412.737340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3412.751082] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.24/24@tcp [ 3416.280796] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3423.203036] LustreError: 127013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3423.244207] LustreError: 127013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3423.882793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3424.146738] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3424.171175] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.24/24@tcp [ 3424.191757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3427.451480] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3433.611822] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3438.338985] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3444.001885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3444.273286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3449.559231] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3455.852636] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3457.605161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3472.868023] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3472.874716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3472.890961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3472.895154] Lustre: Skipped 1 previous similar message [ 3472.906822] Lustre: Skipped 1 previous similar message [ 3477.602507] Lustre: server umount lustre-OST0000 complete [ 3483.106949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3483.112271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3483.124043] Lustre: Skipped 1 previous similar message [ 3483.138790] Lustre: Skipped 1 previous similar message [ 3488.232981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3488.239131] Lustre: Skipped 1 previous similar message [ 3493.350254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3493.361972] Lustre: Skipped 1 previous similar message [ 3495.391122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3495.589753] Lustre: server umount lustre-MDT0000 complete [ 3498.464424] LustreError: 127014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3498.475653] LustreError: 127014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3499.261084] LustreError: 126992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630579 with bad export cookie 11118476889055900222 [ 3499.268382] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3499.282873] LustreError: 126992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3499.593307] Lustre: server umount lustre-MDT0001 complete [ 3509.450555] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 3512.304184] Key type lgssc unregistered [ 3512.620917] LNet: 130608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3512.625724] LNetError: 130608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3512.669180] LNet: Removed LNI 192.168.203.124@tcp [ 3513.508421] Key type .llcrypt unregistered [ 3513.510848] Key type ._llcrypt unregistered [ 3537.128309] Key type ._llcrypt registered [ 3537.131424] Key type .llcrypt registered [ 3537.261686] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 3550.571482] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3551.284229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3551.297354] alg: No test for adler32 (adler32-zlib) [ 3552.280320] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 3552.509098] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 3554.167651] Key type lgssc registered [ 3554.964200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3591.958806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3592.048238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.262144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3593.284409] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3593.335277] Lustre: lustre-MDT0000: new disk, initializing [ 3593.426103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3593.446577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3597.488901] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3606.227463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.334552] Lustre: 134001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3606.387207] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3606.392284] Lustre: Skipped 1 previous similar message [ 3606.461330] Lustre: lustre-MDT0001: new disk, initializing [ 3606.538765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3606.565424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3606.579568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3610.066979] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3617.411121] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3621.687679] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3627.921464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3628.338404] Lustre: lustre-OST0000: new disk, initializing [ 3628.343822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3628.350448] Lustre: 135969:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3628.445202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.278907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3634.286200] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3634.402929] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3635.397224] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3644.925365] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3648.363169] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3648.585080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3652.598876] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3652.759792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3655.136050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3655.139791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.141242] LustreError: Skipped 1 previous similar message [ 3655.147944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3655.154278] Lustre: Skipped 1 previous similar message [ 3660.256307] Lustre: server umount lustre-OST0000 complete [ 3665.377248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3665.379662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3665.402194] Lustre: Skipped 2 previous similar messages [ 3669.377873] Lustre: server umount lustre-MDT0000 complete [ 3672.682846] LustreError: 133993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630753 with bad export cookie 4493417789439169310 [ 3672.687604] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3672.702078] LustreError: 133993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3673.037112] Lustre: server umount lustre-MDT0001 complete [ 3679.745516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3680.047793] LustreError: 138183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3680.075312] LustreError: 138183:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3680.142843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3683.226284] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3685.349661] LustreError: 138182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3689.210718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3689.434751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3693.080535] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3702.639710] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3710.100964] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3717.092358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3717.501851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3722.684583] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3729.366476] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3733.469933] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3733.791424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3738.412652] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3738.618475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3746.938155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3747.108229] Lustre: lustre-OST0001: new disk, initializing [ 3747.111928] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3747.117735] Lustre: 141552:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3747.227030] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3752.699064] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3753.159573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3753.176970] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3753.311781] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3762.255061] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3762.519339] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3767.810414] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3768.248390] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3774.313917] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e579009d800.ost_server_uuid 50 [ 3775.744768] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e579009d800.ost_server_uuid in FULL state after 0 sec [ 3784.164037] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3784.165221] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3784.181536] LustreError: Skipped 1 previous similar message [ 3784.195768] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3784.198500] Lustre: Skipped 2 previous similar messages [ 3785.779182] Lustre: server umount lustre-OST0001 complete [ 3789.282372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3789.284608] LustreError: 140005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3789.285321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3789.296202] Lustre: Skipped 2 previous similar messages [ 3789.317717] LustreError: 140005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3794.414075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3794.416634] LustreError: 140006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3794.424515] Lustre: Skipped 2 previous similar messages [ 3794.471611] LustreError: 140006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3795.281851] Lustre: server umount lustre-OST0000 complete [ 3799.520679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3804.639869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3804.646034] Lustre: Skipped 3 previous similar messages [ 3805.738542] Lustre: server umount lustre-MDT0000 complete [ 3808.632536] LustreError: 138165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630889 with bad export cookie 4493417789439170192 [ 3808.634204] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3808.643841] LustreError: 138165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3808.873964] Lustre: server umount lustre-MDT0001 complete [ 3825.372967] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 3828.014497] Key type lgssc unregistered [ 3828.262527] LNet: 144555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3828.270366] LNetError: 144555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3828.282850] LNet: Removed LNI 192.168.203.124@tcp [ 3828.821493] Key type .llcrypt unregistered [ 3828.823574] Key type ._llcrypt unregistered [ 3862.009808] Key type ._llcrypt registered [ 3862.012540] Key type .llcrypt registered [ 3862.093464] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 3876.145668] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3876.831738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3876.852391] alg: No test for adler32 (adler32-zlib) [ 3877.857556] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 3878.041827] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 3879.689118] Key type lgssc registered [ 3880.717089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3909.167756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3909.202719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.440338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3910.463701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3910.557260] Lustre: lustre-MDT0000: new disk, initializing [ 3910.667982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.700038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3914.924188] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3924.685512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3924.816744] Lustre: 148709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3924.871810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3924.876951] Lustre: Skipped 1 previous similar message [ 3924.956596] Lustre: lustre-MDT0001: new disk, initializing [ 3925.106769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3925.157552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3925.170227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3929.768670] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3939.103575] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3945.212610] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3952.721847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3953.180660] Lustre: lustre-OST0000: new disk, initializing [ 3953.189503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3953.202843] Lustre: 150675:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3953.328516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3959.377476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3959.387214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3959.505643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3960.163729] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3969.098815] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3974.412668] Lustre: Failing over lustre-OST0000 [ 3974.609709] Lustre: server umount lustre-OST0000 complete [ 3990.496143] Lustre: 146934:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786631055/real 1786631055] req@00000000d5faf41b x1873418343449728/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786631071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3990.506501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3990.571293] Lustre: 146934:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3991.582095] Lustre: 146934:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786631056/real 1786631056] req@000000003dccf203 x1873418343450240/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786631072 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3991.609891] Lustre: 146934:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3994.710858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3995.543418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3995.557908] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3996.521712] Lustre: 146934:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786631060/real 1786631060] req@00000000f21c6af2 x1873418343451264/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786631076 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4000.637930] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4001.695515] Lustre: 146932:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786631066/real 1786631066] req@00000000fef8be5c x1873418343452416/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786631082 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4001.716977] Lustre: 146932:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4002.929641] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4003.724762] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4003.735828] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4006.879271] Lustre: 146932:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786631071/real 1786631071] req@000000002b00df05 x1873418343453568/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786631087 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4006.929797] Lustre: 146932:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4007.604154] Lustre: Failing over lustre-MDT0000 [ 4008.196947] Lustre: server umount lustre-MDT0000 complete [ 4010.980606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.002358] Lustre: Skipped 2 previous similar messages [ 4013.918213] LustreError: 150184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4013.941966] LustreError: 150184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4016.099232] LustreError: 148719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4016.138338] LustreError: 148719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4019.051169] LustreError: 150963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.220656] LustreError: 148716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.242328] LustreError: 148716:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4025.284634] LustreError: 150184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4025.324465] LustreError: 150184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4027.423183] Lustre: 146935:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786631091/real 1786631091] req@00000000bdbe4c8d x1873418343463168/t0(0) o400->MGC192.168.203.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786631107 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4027.457341] Lustre: 146935:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4027.465242] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4029.164989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.425071] LustreError: 150184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.464032] LustreError: 150184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4038.113284] LustreError: 146931:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000922c00ad x1873418343468672/t0(0) o250->MGC192.168.203.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4038.284589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4038.334367] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4040.539497] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4041.741477] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4043.768504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4043.773711] Lustre: Skipped 1 previous similar message [ 4043.789142] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4043.847055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4053.991522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4053.996822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4054.015217] Lustre: Skipped 1 previous similar message [ 4054.045455] Lustre: Skipped 1 previous similar message [ 4059.109606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4059.116828] Lustre: Skipped 1 previous similar message [ 4064.226129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4064.235195] Lustre: Skipped 1 previous similar message [ 4065.247307] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4065.385460] Lustre: server umount lustre-OST0000 complete [ 4074.468228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4074.471632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4074.480620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4074.495861] Lustre: Skipped 1 previous similar message [ 4075.879904] Lustre: server umount lustre-MDT0000 complete [ 4079.584576] LustreError: 148719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4079.606199] LustreError: 148719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4079.823527] LustreError: 148700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631160 with bad export cookie 6240585852745121657 [ 4079.827792] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4079.836823] LustreError: 148700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4080.155396] Lustre: server umount lustre-MDT0001 complete [ 4092.808437] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 4095.314183] Key type lgssc unregistered [ 4095.576603] LNet: 154342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4095.588469] LNetError: 154342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4095.604748] LNet: Removed LNI 192.168.203.124@tcp [ 4096.390541] Key type .llcrypt unregistered [ 4096.392712] Key type ._llcrypt unregistered [ 4130.411298] Key type ._llcrypt registered [ 4130.413699] Key type .llcrypt registered [ 4130.535887] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4131.852802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4131.878428] alg: No test for adler32 (adler32-zlib) [ 4132.906667] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 4133.102533] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 4134.799543] Key type lgssc registered [ 4135.908934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4142.337456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4142.349630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4143.666194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4147.565552] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4153.828639] LustreError: 155861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4153.841955] LustreError: 155861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4156.023461] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4158.950523] LustreError: 155862:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.651500] LustreError: 155890:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4159.675060] LustreError: 155890:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4159.687081] LustreError: 155890:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4160.047240] Lustre: server umount lustre-MDT0000 complete [ 4174.911700] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 4186.403765] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4222.748155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4222.929596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4222.960129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4223.020276] Lustre: lustre-MDT0000: new disk, initializing [ 4223.098365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4223.117389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4227.042149] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4236.335561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.542079] Lustre: 159192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4236.641292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4236.644757] Lustre: Skipped 1 previous similar message [ 4236.785317] Lustre: lustre-MDT0001: new disk, initializing [ 4236.888550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4236.951811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4236.968336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4241.121926] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4249.512930] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4254.522385] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4259.315572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4259.504479] Lustre: lustre-OST0000: new disk, initializing [ 4259.507952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4259.511423] Lustre: 161161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4259.605185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.724796] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4268.156698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4268.169418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4268.291934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4275.328397] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4279.774866] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4281.004370] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:29:20 (1786631360) [ 4513.252300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4513.259544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4513.272444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4514.276294] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.281161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4514.297021] Lustre: Skipped 1 previous similar message [ 4514.333544] Lustre: Skipped 2 previous similar messages [ 4516.781042] Lustre: server umount lustre-MDT0000 complete [ 4519.391811] LustreError: 160561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.425567] LustreError: 160561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4520.957650] LustreError: 159186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631601 with bad export cookie 10955105168052683838 [ 4520.965581] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4520.977979] LustreError: 159186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4524.511916] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4524.514503] LustreError: 162814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4524.538682] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4527.483546] Lustre: server umount lustre-MDT0001 complete [ 4537.629699] Lustre: server umount lustre-OST0000 complete [ 4546.385613] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 4555.176557] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4592.334707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4592.602676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4592.632538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4592.685652] Lustre: lustre-MDT0000: new disk, initializing [ 4592.776630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4592.805468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4599.140444] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4610.334069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.409755] Lustre: 165829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4610.524286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4610.533065] Lustre: Skipped 1 previous similar message [ 4610.673190] Lustre: lustre-MDT0001: new disk, initializing [ 4610.823154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4610.896143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4610.905764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4614.699142] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4624.102245] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4629.431673] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4635.934973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4636.295427] Lustre: lustre-OST0000: new disk, initializing [ 4636.299210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4636.304585] Lustre: 167802:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4636.420073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4642.898362] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4644.409705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4644.416832] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4644.517259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4652.600976] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4657.721569] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4657.949881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4662.749443] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4662.942949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4665.312554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4665.322638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4665.330909] LustreError: Skipped 1 previous similar message [ 4665.332355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4665.332362] Lustre: Skipped 1 previous similar message [ 4665.380189] Lustre: Skipped 1 previous similar message [ 4670.437496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4670.442264] Lustre: Skipped 2 previous similar messages [ 4671.112981] Lustre: server umount lustre-OST0000 complete [ 4675.553829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4675.584943] Lustre: Skipped 1 previous similar message [ 4680.675932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4680.682179] Lustre: Skipped 2 previous similar messages [ 4689.375304] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4689.895124] Lustre: server umount lustre-MDT0000 complete [ 4690.925615] LustreError: 167128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4690.950896] LustreError: 167128:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4694.746942] LustreError: 167813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631775 with bad export cookie 10955105168052842668 [ 4694.759409] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4694.771491] LustreError: 167813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4695.255147] Lustre: server umount lustre-MDT0001 complete [ 4704.852475] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:36:23 (1786631783) [ 4723.688503] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 4733.639024] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4773.129717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4773.338481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4773.365436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4773.430081] Lustre: lustre-MDT0000: new disk, initializing [ 4773.503483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4773.543443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4777.818556] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4790.911045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4791.121971] Lustre: 172558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4791.802989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4798.148709] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4808.214394] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4814.319513] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4820.437579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4820.671978] Lustre: lustre-OST0000: new disk, initializing [ 4820.675740] Lustre: Skipped 1 previous similar message [ 4820.680806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4820.684672] Lustre: Skipped 2 previous similar messages [ 4820.689839] Lustre: 174537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4826.912580] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4828.770729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4828.786059] Lustre: Skipped 1 previous similar message [ 4828.789138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4828.879812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4836.369539] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4844.518648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4844.519622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4844.524225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4844.541083] Lustre: Skipped 1 previous similar message [ 4844.556183] Lustre: Skipped 2 previous similar messages [ 4850.218441] Lustre: server umount lustre-OST0000 complete [ 4860.553076] Lustre: server umount lustre-MDT0000 complete [ 4864.167600] LustreError: 172553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631944 with bad export cookie 10955105168052843550 [ 4864.185876] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4864.195849] LustreError: 172553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4864.571668] Lustre: server umount lustre-MDT0001 complete [ 4873.416587] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:39:12 (1786631952) [ 4890.212294] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 4901.913309] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4942.109091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4942.434109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4942.506523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4942.635544] Lustre: lustre-MDT0000: new disk, initializing [ 4942.742104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4942.755571] Lustre: Skipped 2 previous similar messages [ 4942.784407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4948.297302] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4957.981596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4958.161333] Lustre: 178858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4958.415552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4962.643655] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4972.869174] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4979.668425] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4985.854975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4986.019739] Lustre: 180837:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4990.877324] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4996.190758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4996.350735] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4999.681188] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5011.938088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5011.939423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5011.953196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5011.953204] Lustre: Skipped 7 previous similar messages [ 5011.983350] Lustre: Skipped 3 previous similar messages [ 5013.630572] Lustre: server umount lustre-OST0000 complete [ 5031.393614] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5031.571988] Lustre: server umount lustre-MDT0000 complete [ 5035.084187] LustreError: 178851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632115 with bad export cookie 10955105168052844628 [ 5035.098565] LustreError: 178851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5035.098781] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5035.435560] Lustre: server umount lustre-MDT0001 complete [ 5045.366690] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5048.248647] Key type lgssc unregistered [ 5048.539683] LNet: 182696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5048.545505] LNetError: 182696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5048.560754] LNet: Removed LNI 192.168.203.124@tcp [ 5049.499879] Key type .llcrypt unregistered [ 5049.508603] Key type ._llcrypt unregistered [ 5074.155483] Key type ._llcrypt registered [ 5074.159128] Key type .llcrypt registered [ 5074.295342] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5075.233424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5075.398875] alg: No test for adler32 (adler32-zlib) [ 5076.527535] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 5076.734746] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5078.431170] Key type lgssc registered [ 5079.619767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5087.470260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5087.502066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5089.027832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5093.297517] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5098.467577] LustreError: 184113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5098.489891] LustreError: 184113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5101.814171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5102.198432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5106.421132] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5113.443960] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5118.870230] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5125.733088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5126.133976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5132.063339] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5134.329650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5139.635178] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5159.913702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5159.919570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5159.934841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5159.941852] Lustre: Skipped 1 previous similar message [ 5164.261614] Lustre: server umount lustre-OST0000 complete [ 5168.611571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5168.625823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5168.649085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5168.660883] Lustre: Skipped 1 previous similar message [ 5170.146085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5170.148706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5173.744661] Lustre: server umount lustre-MDT0000 complete [ 5175.264315] LustreError: 184841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5175.281740] LustreError: 184841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5176.833229] LustreError: 184094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632257 with bad export cookie 14687313886952635726 [ 5176.848840] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5176.850372] LustreError: 184094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5177.150303] Lustre: server umount lustre-MDT0001 complete [ 5187.811262] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5190.407966] Key type lgssc unregistered [ 5190.709346] LNet: 187671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5190.736085] LNetError: 187671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5190.761317] LNet: Removed LNI 192.168.203.124@tcp [ 5191.885213] Key type .llcrypt unregistered [ 5191.887829] Key type ._llcrypt unregistered [ 5220.355519] Key type ._llcrypt registered [ 5220.357119] Key type .llcrypt registered [ 5220.546382] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5221.920430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5222.040132] alg: No test for adler32 (adler32-zlib) [ 5223.110453] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 5223.546271] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5225.319704] Key type lgssc registered [ 5226.191484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5235.359329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5235.377654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5236.704243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5241.060814] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5246.946489] LustreError: 189092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5246.977282] LustreError: 189092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5249.658958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5250.077139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5254.295888] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5260.860992] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5266.289537] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5277.002230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5277.405603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5281.524544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5285.510078] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5292.904589] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5302.244836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5302.251506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5302.276758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5304.695864] Lustre: lustre-OST0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 5305.324610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5305.342391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5307.980578] Lustre: server umount lustre-OST0000 complete [ 5315.511900] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8e57845c0800.ost_server_uuid 50 [ 5317.067070] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5330.912594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5330.931918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5330.943761] Lustre: Skipped 3 previous similar messages [ 5336.035683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5336.041059] Lustre: Skipped 1 previous similar message [ 5337.020918] Lustre: server umount lustre-MDT0000 complete [ 5340.367541] LustreError: 189072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632421 with bad export cookie 7193337915340427599 [ 5340.371571] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5340.376867] LustreError: 189072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5340.706271] Lustre: server umount lustre-MDT0001 complete [ 5348.900474] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:47:08 (1786632428) [ 5355.257802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5355.721812] LustreError: 193008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5355.796306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5360.762631] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5361.120672] LustreError: 193009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5366.245697] LustreError: 193008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5368.486162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.916492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5373.890718] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5380.971873] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5385.967477] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5392.373545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5392.785249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5398.829289] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5401.017453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5405.690792] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5412.930081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5413.086308] Lustre: lustre-OST0001: new disk, initializing [ 5413.094290] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5413.104170] Lustre: 195849:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5413.224585] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5419.741972] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5421.703597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5421.712921] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5421.884302] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5430.046876] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5434.848592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5434.852477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5434.877063] Lustre: Skipped 1 previous similar message [ 5434.885084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5436.898746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5445.987877] Lustre: lustre-OST0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 5446.002560] Lustre: Skipped 7 previous similar messages [ 5447.143090] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5447.407658] Lustre: server umount lustre-OST0000 complete [ 5447.661570] LustreError: 194828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5447.687964] LustreError: 194828:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5453.940571] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5454.220895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5456.240761] LustreError: 194830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5456.269629] LustreError: 194830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5458.993317] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5459.193662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5473.258819] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5473.264611] LustreError: 195304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5473.280875] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5473.280884] Lustre: Skipped 1 previous similar message [ 5473.289995] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5473.308861] LustreError: 195304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5473.327775] Lustre: Skipped 1 previous similar message [ 5478.868953] Lustre: server umount lustre-OST0001 complete [ 5483.492183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5483.505588] Lustre: Skipped 2 previous similar messages [ 5488.719615] Lustre: server umount lustre-MDT0000 complete [ 5491.849157] LustreError: 192989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632572 with bad export cookie 7193337915340428670 [ 5491.854811] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5491.860233] LustreError: 192989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5492.210311] Lustre: server umount lustre-MDT0001 complete [ 5514.597189] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:49:53 (1786632593) [ 5521.023602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5521.300208] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5521.349780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5521.532428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5525.894949] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5534.656715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5534.739427] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5534.779300] Lustre: 199287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5540.149244] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5547.759599] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5553.143697] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5560.507216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5560.720603] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5566.635598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5566.923124] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5574.314405] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5580.592183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5580.691090] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5586.850424] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5593.747846] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5597.153561] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5597.166677] LustreError: Skipped 1 previous similar message [ 5597.169363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5597.176069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5597.189338] Lustre: Skipped 1 previous similar message [ 5597.197101] Lustre: Skipped 7 previous similar messages [ 5602.352258] Lustre: server umount lustre-OST0000 complete [ 5604.706254] LustreError: 201138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5604.739973] LustreError: 201138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5622.756709] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5622.778686] LustreError: Skipped 1 previous similar message [ 5624.310727] Lustre: server umount lustre-OST0001 complete [ 5632.480958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5632.503769] Lustre: Skipped 3 previous similar messages [ 5642.207172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5642.446587] Lustre: server umount lustre-MDT0000 complete [ 5646.327503] LustreError: 200035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632726 with bad export cookie 7193337915340430077 [ 5646.339279] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5646.354784] LustreError: 200035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5646.997903] Lustre: server umount lustre-MDT0001 complete [ 5668.967788] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:52:27 (1786632747) [ 5685.239448] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 5694.693316] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5733.670562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5733.996572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5734.043603] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5734.188877] Lustre: lustre-MDT0000: new disk, initializing [ 5734.294212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5734.299206] Lustre: Skipped 3 previous similar messages [ 5734.312646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5739.368818] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5750.367436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5750.508343] Lustre: 207387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5750.557357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5750.560337] Lustre: Skipped 1 previous similar message [ 5750.627358] Lustre: lustre-MDT0001: new disk, initializing [ 5750.739428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5750.749666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5754.547165] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5764.256864] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5769.839926] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5776.466084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5776.720553] Lustre: lustre-OST0000: new disk, initializing [ 5776.725639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5776.731994] Lustre: 209354:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5783.137980] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5784.627715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5784.636809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5784.735347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5793.608581] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5799.086292] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5799.357877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5804.509771] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5804.760307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5810.659509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5810.661227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5810.685885] Lustre: Skipped 1 previous similar message [ 5810.711233] Lustre: Skipped 11 previous similar messages [ 5813.318372] Lustre: server umount lustre-OST0000 complete [ 5824.065689] Lustre: server umount lustre-MDT0000 complete [ 5826.016889] LustreError: 207394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5826.045761] LustreError: 207394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 5828.138799] LustreError: 207378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632908 with bad export cookie 7193337915340431638 [ 5828.152123] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5828.157106] LustreError: 207378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5828.427308] Lustre: server umount lustre-MDT0001 complete [ 5837.746965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.431088] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5852.547899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5857.677965] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5865.368297] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5870.630054] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5877.153701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5877.647048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5877.651957] Lustre: Skipped 4 previous similar messages [ 5883.616788] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5890.744540] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5895.305703] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5895.506940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5900.057967] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5900.270451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5904.361416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5904.365294] LustreError: Skipped 1 previous similar message [ 5908.101916] Lustre: server umount lustre-OST0000 complete [ 5914.223261] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5920.502445] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5948.967318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5955.133779] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5962.842122] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5968.640398] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 5969.904232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.915634] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.927340] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5975.009230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5975.023157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5975.030429] Lustre: Skipped 4 previous similar messages [ 5975.036605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5975.043858] LustreError: Skipped 1 previous similar message [ 5975.059605] Lustre: Skipped 5 previous similar messages [ 5987.304397] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5987.515871] Lustre: server umount lustre-OST0000 complete [ 5997.836757] Lustre: server umount lustre-MDT0000 complete [ 6000.612160] LustreError: 212306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6000.644293] LustreError: 212306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6001.394938] LustreError: 211559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633082 with bad export cookie 7193337915340432520 [ 6001.407304] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6001.769953] Lustre: server umount lustre-MDT0001 complete [ 6009.420732] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:58:08 (1786633088) [ 6015.538444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.108287] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6028.025700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6032.856858] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6040.986041] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6047.149407] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6054.527624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6062.224915] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6070.238201] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6075.190540] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6075.455578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6081.494521] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6081.814831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6089.538861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6089.708383] Lustre: lustre-OST0001: new disk, initializing [ 6089.715950] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6089.723832] Lustre: 220915:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6096.457540] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6096.945885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6096.957049] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6097.068784] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6106.271956] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6111.314712] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6111.562674] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6116.242994] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6116.498109] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6124.687729] Lustre: server umount lustre-OST0001 complete [ 6130.942845] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6131.210201] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6135.958817] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6136.228972] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6164.525800] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6164.751728] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6164.758988] Lustre: Skipped 5 previous similar messages [ 6170.095987] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6170.111752] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6170.111803] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6170.119733] Lustre: Skipped 1 previous similar message [ 6170.934540] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6178.006071] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6179.688715] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6185.444154] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6185.453262] LustreError: Skipped 1 previous similar message [ 6188.137405] Lustre: server umount lustre-OST0001 complete [ 6210.845293] Lustre: server umount lustre-OST0000 complete [ 6220.497903] Lustre: server umount lustre-MDT0000 complete [ 6224.155082] LustreError: 224499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633304 with bad export cookie 7193337915340433486 [ 6224.161151] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6224.168422] LustreError: 224499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6230.591848] Lustre: server umount lustre-MDT0001 complete [ 6252.844641] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:02:11 (1786633331) [ 6259.744284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6260.014482] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6260.065934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6264.836747] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6273.345641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6273.477374] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6273.517607] Lustre: 226172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6278.486376] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6286.230197] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6292.356130] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6298.687415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6298.910179] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6305.346707] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6313.869700] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6323.494043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6323.575056] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6330.823021] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6338.758205] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6340.411521] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6345.113208] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6345.354212] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6350.021789] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6350.277268] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6355.811791] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e5785bf6000.ost_server_uuid 50 [ 6357.552100] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e5785bf6000.ost_server_uuid in IDLE state after 0 sec [ 6358.681984] Lustre: Permanently deactivating lustre-OST0001 [ 6358.707354] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6362.508426] Lustre: Permanently reactivating lustre-OST0001 [ 6368.224636] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6368.233239] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6368.251107] Lustre: Skipped 12 previous similar messages [ 6368.263844] Lustre: Skipped 17 previous similar messages [ 6369.263917] Lustre: server umount lustre-OST0001 complete [ 6373.365803] LustreError: 228015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6373.387456] LustreError: 228015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [ 6378.702466] Lustre: server umount lustre-OST0000 complete [ 6388.706203] Lustre: server umount lustre-MDT0000 complete [ 6392.228831] LustreError: 226164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633472 with bad export cookie 7193337915340434774 [ 6392.231097] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6392.246221] LustreError: 226164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6392.734128] Lustre: server umount lustre-MDT0001 complete [ 6416.961144] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:04:55 (1786633495) [ 6425.583048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6426.066876] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6426.120137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6426.130582] Lustre: Skipped 5 previous similar messages [ 6432.240530] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6443.111543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6443.165134] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6443.187792] Lustre: 232861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6448.179743] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6456.368713] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6462.127821] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6469.175598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6469.378117] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6469.401839] Lustre: Permanently deactivating lustre-OST0000 [ 6469.415902] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6475.452963] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6476.896899] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6482.534460] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6488.523323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6488.631727] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6495.400591] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6495.570971] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6501.945130] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6503.599032] Lustre: Permanently reactivating lustre-OST0000 [ 6503.627746] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6503.639651] Lustre: Skipped 2 previous similar messages [ 6511.157277] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6511.157293] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 34 seconds. I think it's dead, and I am evicting it. exp 00000000c90986a4, cur 1786633591 deadline 1786633587 last 1786633557 [ 6511.165844] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6511.168654] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6511.220605] Lustre: Skipped 1 previous similar message [ 6511.257049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6521.203893] Lustre: server umount lustre-OST0001 complete [ 6521.312624] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6521.322788] LustreError: Skipped 4 previous similar messages [ 6530.733794] Lustre: server umount lustre-OST0000 complete [ 6540.622759] Lustre: server umount lustre-MDT0000 complete [ 6543.762776] LustreError: 232853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633624 with bad export cookie 7193337915340436454 [ 6543.776731] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6544.046930] Lustre: server umount lustre-MDT0001 complete [ 6554.829257] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6557.878953] Key type lgssc unregistered [ 6558.126054] LNet: 237742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6558.144755] LNetError: 237742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6558.160132] LNet: Removed LNI 192.168.203.124@tcp [ 6558.889348] Key type .llcrypt unregistered [ 6558.893879] Key type ._llcrypt unregistered [ 6580.940296] Key type ._llcrypt registered [ 6580.943291] Key type .llcrypt registered [ 6581.028967] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6581.777799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6581.852214] alg: No test for adler32 (adler32-zlib) [ 6582.922881] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 6583.102988] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6584.855199] Key type lgssc registered [ 6586.009468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6595.002414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6595.040047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6596.467892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6601.252703] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6606.816338] LustreError: 239210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6606.843636] LustreError: 239210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6609.245738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6609.358542] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6609.385808] Lustre: 239197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6609.398471] Lustre: Permanently deactivating lustre-MDT0001 [ 6609.419855] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6609.777844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6610.779252] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6610.788579] LustreError: 239241:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6610.804726] LustreError: 239241:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6610.815795] LustreError: 239241:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 1, failed: rc = -108 [ 6614.248287] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6621.169694] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6625.673338] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6631.455101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6631.763725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6634.854672] LustreError: 241051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6634.868650] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6637.294976] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6640.111770] LustreError: 241053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6642.154939] LustreError: 241051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6642.170719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6642.174415] LustreError: 241051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6643.948596] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6647.269141] LustreError: 241051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6647.292277] LustreError: 241051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6650.592345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6650.756731] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6655.982082] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6655.998806] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6656.709272] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6663.826245] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6666.726224] Lustre: Permanently reactivating lustre-MDT0001 [ 6666.747179] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6666.759114] Lustre: Skipped 2 previous similar messages [ 6672.312690] Lustre: lustre-MDT0001: Client 4eb886f5-005f-4b90-b398-3fe87639c6f8 (at 192.168.203.24@tcp) reconnecting [ 6678.937995] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6680.628764] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6685.911431] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6686.191383] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6690.531694] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6690.802865] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6693.356272] Lustre: Permanently deactivating lustre-MDT0001 [ 6693.373637] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6693.380379] Lustre: Skipped 2 previous similar messages [ 6699.123034] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6707.168045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6707.176280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.178971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6707.219474] Lustre: Skipped 2 previous similar messages [ 6707.379336] Lustre: server umount lustre-MDT0000 complete [ 6711.231206] LustreError: 243692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633791 with bad export cookie 6074615961662397641 [ 6711.240556] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6711.248766] LustreError: 243692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6711.626351] Lustre: server umount lustre-MDT0001 complete [ 6717.596848] Lustre: server umount lustre-OST0000 complete [ 6728.797934] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:10:07 (1786633807) [ 6740.272915] Lustre: server umount lustre-OST0001 complete [ 6747.383513] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 6757.131361] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6791.159546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6791.377298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6791.387062] Lustre: Skipped 2 previous similar messages [ 6791.416724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6791.502415] Lustre: lustre-MDT0000: new disk, initializing [ 6791.578541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6791.590656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6795.746951] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6805.214222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6805.380376] Lustre: 247467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6805.416449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6805.419515] Lustre: Skipped 1 previous similar message [ 6805.481405] Lustre: lustre-MDT0001: new disk, initializing [ 6805.575345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6805.620575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6805.634209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6809.196029] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6817.268380] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6822.032001] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6828.973441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6829.269647] Lustre: lustre-OST0000: new disk, initializing [ 6829.276643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6829.283243] Lustre: 249435:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6829.342987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6835.217477] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6838.311635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6838.324306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6838.451932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6844.175587] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6846.264674] LustreError: 247476:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6854.696450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6854.864566] Lustre: lustre-OST0001: new disk, initializing [ 6854.867640] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6854.873131] Lustre: 250552:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6856.296522] LustreError: 247476:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6856.331487] LustreError: 248892:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6861.690351] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6864.487987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6864.496164] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6864.580906] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6866.367148] LustreError: 248892:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6871.448566] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6873.069542] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6875.118628] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6875.119207] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6875.127190] LustreError: Skipped 1 previous similar message [ 6875.128501] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6875.128507] Lustre: Skipped 3 previous similar messages [ 6875.160361] Lustre: Skipped 2 previous similar messages [ 6880.231732] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6880.236035] Lustre: Skipped 1 previous similar message [ 6881.198256] Lustre: server umount lustre-OST0001 complete [ 6885.350306] LustreError: 249420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.378368] LustreError: 249420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6899.683905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6899.689201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6899.705995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6903.874643] Lustre: server umount lustre-OST0000 complete [ 6907.872064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6907.877516] LustreError: Skipped 1 previous similar message [ 6907.882881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6907.892895] Lustre: Skipped 1 previous similar message [ 6907.899642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6907.912659] Lustre: Skipped 2 previous similar messages [ 6914.063578] Lustre: server umount lustre-MDT0000 complete [ 6916.066505] LustreError: 247479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6916.078348] LustreError: 247479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6917.448883] LustreError: 247460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633998 with bad export cookie 6074615961662400448 [ 6917.454923] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6917.456182] LustreError: 247460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6924.008758] Lustre: server umount lustre-MDT0001 complete [ 6935.843952] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6938.478257] Key type lgssc unregistered [ 6938.849550] LNet: 252590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6938.855181] LNetError: 252590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6938.901513] LNet: Removed LNI 192.168.203.124@tcp [ 6939.762043] Key type .llcrypt unregistered [ 6939.770148] Key type ._llcrypt unregistered [ 6979.350609] Key type ._llcrypt registered [ 6979.353065] Key type .llcrypt registered [ 6979.498686] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6981.153439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6981.217949] alg: No test for adler32 (adler32-zlib) [ 6982.471877] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 6982.776928] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6984.479166] Key type lgssc registered [ 6985.798251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6993.708538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6993.727683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6994.937449] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6994.972184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6995.147642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6999.927774] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7008.655512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7008.779749] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7008.862728] Lustre: 254798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7009.118659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7013.231228] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7021.127376] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7027.318905] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7034.426971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7034.683940] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7034.883824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7041.491692] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7044.756516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7049.412624] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7100.900093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7100.906576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7100.919331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7101.412927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7101.418697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.425517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7106.324334] Lustre: server umount lustre-OST0000 complete [ 7110.438795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7122.649678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7123.039334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7127.151502] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7127.164628] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7129.376205] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7138.034398] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7141.555745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7141.573798] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7177.704960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7177.706769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7177.715986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7177.715994] Lustre: Skipped 1 previous similar message [ 7177.742182] Lustre: Skipped 1 previous similar message [ 7182.816370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7182.831919] Lustre: Skipped 2 previous similar messages [ 7183.590872] Lustre: server umount lustre-OST0000 complete [ 7187.937630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7187.939671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7187.956351] Lustre: Skipped 1 previous similar message [ 7198.179706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7198.191389] Lustre: Skipped 4 previous similar messages [ 7201.247144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7201.424209] Lustre: server umount lustre-MDT0000 complete [ 7204.567559] LustreError: 254791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634285 with bad export cookie 7552952524182257576 [ 7204.570766] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7204.584370] LustreError: 254791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7204.767261] Lustre: server umount lustre-MDT0001 complete [ 7214.959739] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7218.103169] Key type lgssc unregistered [ 7218.428521] LNet: 260225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7218.447193] LNetError: 260225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7218.470262] LNet: Removed LNI 192.168.203.124@tcp [ 7219.352494] Key type .llcrypt unregistered [ 7219.355369] Key type ._llcrypt unregistered [ 7242.190466] Key type ._llcrypt registered [ 7242.193552] Key type .llcrypt registered [ 7242.311296] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 7243.086074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7243.148639] alg: No test for adler32 (adler32-zlib) [ 7244.182634] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 7244.463194] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 7246.151407] Key type lgssc registered [ 7247.373646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7254.336954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7254.374074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7255.755697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7260.414500] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7266.281932] LustreError: 261642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7266.297020] LustreError: 261642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7267.301897] LustreError: 261643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7268.617813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7268.892892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7274.189393] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7281.875278] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7286.667646] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7293.435963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7293.805894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7300.115782] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7301.999712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7307.832881] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7337.957511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7337.967073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7337.990728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7340.006198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7340.011645] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7344.178728] Lustre: server umount lustre-OST0000 complete [ 7350.240752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7350.247945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7350.256953] Lustre: Skipped 1 previous similar message [ 7350.270737] Lustre: Skipped 1 previous similar message [ 7353.907414] Lustre: server umount lustre-MDT0000 complete [ 7355.360553] LustreError: 262371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7355.392083] LustreError: 262371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7357.570487] LustreError: 261623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634438 with bad export cookie 10539009146387509525 [ 7357.572605] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7357.589573] LustreError: 261623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7358.010840] Lustre: server umount lustre-MDT0001 complete [ 7368.950466] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7371.787945] Key type lgssc unregistered [ 7372.176180] LNet: 265971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7372.182557] LNetError: 265971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7372.201662] LNet: Removed LNI 192.168.203.124@tcp [ 7373.157187] Key type .llcrypt unregistered [ 7373.162186] Key type ._llcrypt unregistered [ 7392.124249] Key type ._llcrypt registered [ 7392.128170] Key type .llcrypt registered [ 7392.229992] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 7393.144110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7393.186852] alg: No test for adler32 (adler32-zlib) [ 7394.255593] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 7394.530809] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 7396.215305] Key type lgssc registered [ 7397.405235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7404.324141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7404.351920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7405.844460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7409.843650] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7416.292689] LustreError: 267196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7416.316741] LustreError: 267196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7417.450964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7417.756104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7422.248603] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7429.231455] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7434.114079] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7439.810184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7440.179241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7445.824645] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7450.425562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7453.094733] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7463.905456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7463.911069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.920742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7465.957262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7465.965252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7465.978755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7465.983366] Lustre: Skipped 1 previous similar message [ 7467.867252] Lustre: server umount lustre-OST0000 complete [ 7474.146439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7474.154919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7474.181383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7476.193961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7476.194085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7477.873372] Lustre: server umount lustre-MDT0000 complete [ 7481.006047] LustreError: 270120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634561 with bad export cookie 809696406163101363 [ 7481.013492] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7481.024316] LustreError: 270120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7481.350732] Lustre: server umount lustre-MDT0001 complete [ 7491.482604] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7493.955177] Key type lgssc unregistered [ 7494.230730] LNet: 270875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7494.239644] LNetError: 270875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7494.261736] LNet: Removed LNI 192.168.203.124@tcp [ 7494.972396] Key type .llcrypt unregistered [ 7494.979519] Key type ._llcrypt unregistered [ 7518.542831] Key type ._llcrypt registered [ 7518.546260] Key type .llcrypt registered [ 7518.643129] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 7519.663816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7519.726391] alg: No test for adler32 (adler32-zlib) [ 7520.795670] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 7520.983702] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 7522.623187] Key type lgssc registered [ 7523.742762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7531.039992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7531.063722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7532.538192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7536.787708] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7542.760372] LustreError: 272292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7542.787617] LustreError: 272292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7544.093874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7544.348216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7548.641730] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7555.410526] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7560.990354] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7567.771772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7568.209065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7573.655351] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7577.451231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7580.185884] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7600.096226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7600.103416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7600.128408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7603.171326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7603.180227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7603.209568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7606.308891] Lustre: server umount lustre-OST0000 complete [ 7610.853496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7610.858077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7610.868680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7610.870589] Lustre: Skipped 1 previous similar message [ 7613.409545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7613.413038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7616.625038] Lustre: server umount lustre-MDT0000 complete [ 7618.529512] LustreError: 273020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7618.561925] LustreError: 273020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7620.833176] LustreError: 272272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634701 with bad export cookie 17461900893775638503 [ 7620.848908] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7620.854041] LustreError: 272272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7621.269353] Lustre: server umount lustre-MDT0001 complete [ 7631.818050] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7634.248449] Key type lgssc unregistered [ 7634.611567] LNet: 276600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7634.619507] LNetError: 276600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7634.637680] LNet: Removed LNI 192.168.203.124@tcp [ 7635.674096] Key type .llcrypt unregistered [ 7635.681054] Key type ._llcrypt unregistered [ 7654.589229] Key type ._llcrypt registered [ 7654.598611] Key type .llcrypt registered [ 7654.707527] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 7656.112456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7656.156336] alg: No test for adler32 (adler32-zlib) [ 7657.263404] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 7657.497090] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 7659.207477] Key type lgssc registered [ 7660.408698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7668.673395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7668.715302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7670.187319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7674.423357] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7679.461869] LustreError: 277838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7679.487803] LustreError: 277838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7683.679427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7684.165461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7689.540829] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7699.144781] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7706.149492] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7713.523688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7713.941351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7715.018126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7720.412477] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7727.966970] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7739.878175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7739.892547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7739.898869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7739.926757] Lustre: Skipped 1 previous similar message [ 7744.995616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7744.998379] Lustre: Skipped 1 previous similar message [ 7746.008433] Lustre: server umount lustre-OST0000 complete [ 7750.625031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7750.635580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7750.667500] Lustre: Skipped 1 previous similar message [ 7750.673918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7755.233908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7755.234434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7755.264169] Lustre: Skipped 1 previous similar message [ 7756.572834] Lustre: server umount lustre-MDT0000 complete [ 7760.352518] LustreError: 277851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7760.382121] LustreError: 277851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7760.892333] LustreError: 277802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634841 with bad export cookie 2021246233763664104 [ 7760.897802] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7760.905174] LustreError: 277802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7761.256795] Lustre: server umount lustre-MDT0001 complete [ 7773.406475] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7776.709624] Key type lgssc unregistered [ 7777.016944] LNet: 281543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7777.021403] LNetError: 281543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7777.071920] LNet: Removed LNI 192.168.203.124@tcp [ 7778.002351] Key type .llcrypt unregistered [ 7778.005111] Key type ._llcrypt unregistered [ 7790.671317] Key type ._llcrypt registered [ 7790.673881] Key type .llcrypt registered [ 7790.840349] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7810.020546] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 7825.002923] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 7825.777743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7825.905341] alg: No test for adler32 (adler32-zlib) [ 7826.986284] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 7827.375819] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 7829.119168] Key type lgssc registered [ 7829.971910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7863.948690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7863.991823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7865.322898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7865.343651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7865.407409] Lustre: lustre-MDT0000: new disk, initializing [ 7865.469078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7865.484626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7870.175908] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7882.686361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.809922] Lustre: 285100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7882.846743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7882.849889] Lustre: Skipped 1 previous similar message [ 7882.903673] Lustre: lustre-MDT0001: new disk, initializing [ 7882.963219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7882.995368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7883.004067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7887.740263] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7898.484820] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7904.479851] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7912.008923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7912.338683] Lustre: lustre-OST0000: new disk, initializing [ 7912.341750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7912.350886] Lustre: 287067:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7912.445094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7918.163057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7918.176901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7918.313778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7919.157853] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7929.202677] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7933.987421] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7934.239431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7938.866709] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7939.068637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7944.163541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7944.166440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7944.199437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7947.159506] Lustre: server umount lustre-OST0000 complete [ 7954.402218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7954.419551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7954.422650] Lustre: Skipped 2 previous similar messages [ 7954.433988] Lustre: Skipped 2 previous similar messages [ 7957.428840] Lustre: server umount lustre-MDT0000 complete [ 7961.009666] LustreError: 285093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635041 with bad export cookie 3475977456663237378 [ 7961.011321] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7961.021841] LustreError: 285093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7961.309359] Lustre: server umount lustre-MDT0001 complete [ 7970.035346] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:30:48 (1786635048) [ 7976.343367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7976.689404] LustreError: 289477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7976.705725] LustreError: 289477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7976.755070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7980.986834] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7982.051639] LustreError: 289478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7987.172137] LustreError: 289477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7989.008503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7989.434436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7993.690830] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8000.868243] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8007.371280] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8015.094425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8015.476821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8021.582501] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8028.932599] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8042.471279] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8042.478931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8042.499032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8044.516386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8044.522934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8047.702817] Lustre: server umount lustre-OST0000 complete [ 8054.754380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8054.757763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8054.776258] Lustre: Skipped 1 previous similar message [ 8054.790970] Lustre: Skipped 2 previous similar messages [ 8057.686948] Lustre: server umount lustre-MDT0000 complete [ 8059.875414] LustreError: 289477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8059.890773] LustreError: 289477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8061.089120] LustreError: 289459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635141 with bad export cookie 3475977456663238267 [ 8061.096879] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8061.099270] LustreError: 289459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8061.785869] Lustre: server umount lustre-MDT0001 complete [ 8072.582864] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 8075.725417] Key type lgssc unregistered [ 8076.242734] LNet: 293028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8076.252262] LNetError: 293028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8076.276224] LNet: Removed LNI 192.168.203.124@tcp [ 8077.344337] Key type .llcrypt unregistered [ 8077.347264] Key type ._llcrypt unregistered [ 8103.210595] Key type ._llcrypt registered [ 8103.215568] Key type .llcrypt registered [ 8103.331150] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 8118.747676] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 8120.244799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8120.441782] alg: No test for adler32 (adler32-zlib) [ 8121.571773] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 8121.837570] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 8123.535177] Key type lgssc registered [ 8125.156214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8173.987385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8174.043135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8175.239823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8175.270317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8175.372521] Lustre: lustre-MDT0000: new disk, initializing [ 8175.460279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8175.478409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8179.513356] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8189.418993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8189.584309] Lustre: 296860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8189.696916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8189.708297] Lustre: Skipped 1 previous similar message [ 8189.799299] Lustre: lustre-MDT0001: new disk, initializing [ 8189.871464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8189.915311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8189.941382] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8194.865915] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8204.020324] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8209.461821] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8216.298199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8216.582814] Lustre: lustre-OST2710: new disk, initializing [ 8216.590603] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8216.597779] Lustre: 298827:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8216.707876] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8223.235343] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8223.794341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8223.803468] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8223.931616] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8232.711886] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8239.202456] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8239.381480] Lustre: lustre-OST03e8: new disk, initializing [ 8239.384601] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8239.390393] Lustre: 299890:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8239.469234] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8245.063927] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8245.332298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8245.345976] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8245.473257] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8252.263699] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8258.653060] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8258.842090] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8263.260688] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8263.525129] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8267.954593] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8268.175618] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8272.131809] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8272.287745] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8276.450686] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8276.456295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8276.469775] Lustre: Skipped 3 previous similar messages [ 8276.491630] Lustre: Skipped 3 previous similar messages [ 8281.539854] Lustre: server umount lustre-MDT0000 complete [ 8285.686445] LustreError: 298826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635366 with bad export cookie 13959109042578630172 [ 8285.695840] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8285.703758] LustreError: 298826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8286.252886] Lustre: server umount lustre-MDT0001 complete [ 8300.926150] Lustre: server umount lustre-OST2710 complete [ 8303.072251] Lustre: 294654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786635367/real 1786635367] req@00000000e37e3435 x1873422793637760/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786635383 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8303.105665] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8305.251609] Lustre: server umount lustre-OST03e8 complete [ 8323.307445] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 8333.098343] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 8369.433435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8369.658781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8369.691356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8369.758505] Lustre: lustre-MDT0000: new disk, initializing [ 8369.820435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8369.841572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8375.994953] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8387.019437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8387.144290] Lustre: 305134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8387.190525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8387.195238] Lustre: Skipped 1 previous similar message [ 8387.258563] Lustre: lustre-MDT0001: new disk, initializing [ 8387.342372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8387.389619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8387.409440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8392.268535] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8402.225765] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8408.374813] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8415.153351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8415.439238] Lustre: lustre-OST0000: new disk, initializing [ 8415.446166] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8415.457787] Lustre: 307105:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8415.553164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8421.439594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8421.449337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8421.572258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8421.828198] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8430.781726] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8435.090469] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8435.274193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8439.453410] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8439.647706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8442.336425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8442.337597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8442.342755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8442.357456] Lustre: Skipped 1 previous similar message [ 8447.474419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8447.487943] Lustre: Skipped 2 previous similar messages [ 8447.698180] Lustre: server umount lustre-OST0000 complete [ 8452.577306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8452.597604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8452.605624] Lustre: Skipped 1 previous similar message [ 8457.696872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8457.701872] Lustre: Skipped 1 previous similar message [ 8465.376096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8465.588683] Lustre: server umount lustre-MDT0000 complete [ 8467.939364] LustreError: 305147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8467.954428] LustreError: 305147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8469.191363] LustreError: 305129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635549 with bad export cookie 13959109042578631859 [ 8469.193198] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8469.204789] LustreError: 305129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8469.556322] Lustre: server umount lustre-MDT0001 complete [ 8478.043139] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:39:17 (1786635557) [ 8479.499896] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8481.164633] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:39:20 (1786635560) [ 8484.769794] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 8512.152925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8512.577869] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8512.606432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8512.877624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8517.768148] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8526.322552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8526.438331] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8526.486411] Lustre: 310967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8532.556662] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8555.692188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8555.880781] Lustre: Denying initial registration attempt from nid 192.168.203.124@tcp, specified as failover [ 8555.886491] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8555.900512] LustreError: 312423:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8555.909184] LustreError: 312423:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8555.915758] LustreError: 312423:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8556.009269] Lustre: server umount lustre-OST0000 complete [ 8556.013739] LustreError: 312423:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8561.632442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8561.643558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8561.644996] Lustre: Skipped 2 previous similar messages [ 8561.653912] Lustre: Skipped 3 previous similar messages [ 8567.958085] Lustre: server umount lustre-MDT0000 complete [ 8571.231126] LustreError: 310959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635651 with bad export cookie 13959109042578632741 [ 8571.239234] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8571.241476] LustreError: 310959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8571.456757] Lustre: server umount lustre-MDT0001 complete [ 8581.703477] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 8584.539494] Key type lgssc unregistered [ 8584.900341] LNet: 313546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8584.905611] LNetError: 313546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8584.932411] LNet: Removed LNI 192.168.203.124@tcp [ 8585.840255] Key type .llcrypt unregistered [ 8585.844174] Key type ._llcrypt unregistered [ 8597.057846] Key type ._llcrypt registered [ 8597.061246] Key type .llcrypt registered [ 8597.158172] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 8598.122112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8598.294741] alg: No test for adler32 (adler32-zlib) [ 8599.418366] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 8599.659338] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 8601.319209] Key type lgssc registered [ 8602.497331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8650.060904] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 8657.866859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8657.896365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8659.227300] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8659.268684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8659.429537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8664.737313] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8673.885959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8674.044566] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8674.094339] Lustre: 316309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8674.372764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8678.444572] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8686.331768] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8691.699895] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8711.149935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8711.310547] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8711.448291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8717.604911] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8725.539070] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8728.548457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8728.558134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8728.578127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8729.075872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8729.077696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8729.082633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8733.808992] Lustre: server umount lustre-OST0000 complete [ 8739.299572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8739.309409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8739.312178] Lustre: Skipped 1 previous similar message [ 8739.327906] Lustre: Skipped 2 previous similar messages [ 8743.594298] Lustre: server umount lustre-MDT0000 complete [ 8747.189473] LustreError: 316303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635827 with bad export cookie 688546998217768291 [ 8747.192640] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8747.201909] LustreError: 316303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8747.583266] Lustre: server umount lustre-MDT0001 complete [ 8758.863530] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 8761.670926] Key type lgssc unregistered [ 8761.941767] LNet: 319887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8761.957201] LNetError: 319887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8761.980808] LNet: Removed LNI 192.168.203.124@tcp [ 8762.943071] Key type .llcrypt unregistered [ 8762.946362] Key type ._llcrypt unregistered [ 8787.962371] Key type ._llcrypt registered [ 8787.964180] Key type .llcrypt registered [ 8788.078499] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 8789.235896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8789.275157] alg: No test for adler32 (adler32-zlib) [ 8790.403922] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 8790.694738] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 8792.399186] Key type lgssc registered [ 8793.957531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8802.223496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8802.248334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8803.834690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8808.391655] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8814.052406] LustreError: 321307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8814.079504] LustreError: 321307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8817.053994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8817.440243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8821.896134] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8829.282959] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8835.988816] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8844.275030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8844.710353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8851.327202] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8859.648287] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8873.953764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8873.970997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8873.989086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8876.924187] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 8877.538230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8877.548262] Lustre: Skipped 1 previous similar message [ 8882.014298] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 8882.024753] Lustre: Skipped 3 previous similar messages [ 8887.142360] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 8887.150631] Lustre: Skipped 3 previous similar messages [ 8887.263251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8887.517428] Lustre: server umount lustre-MDT0000 complete [ 8887.782030] LustreError: 322342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8891.563427] LustreError: 321287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635972 with bad export cookie 18074220983157581419 [ 8891.588472] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8892.262710] LustreError: 321304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8892.298855] LustreError: 321304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8892.302256] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.24@tcp (stopping) [ 8892.898361] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8896.947058] LustreError: 321303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8896.980181] LustreError: 321303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8898.198165] Lustre: server umount lustre-MDT0001 complete [ 8904.790632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8912.729282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8916.823629] LustreError: 323139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8916.840273] LustreError: 323139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8916.890940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8916.952606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8921.340191] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8927.080372] LustreError: 324918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8927.098090] LustreError: 324918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8927.144567] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8929.412228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8929.809619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8934.166692] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8934.906139] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8941.636157] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8943.163428] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8948.627887] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8950.042313] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8955.366145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8955.374764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8955.378685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8955.378693] Lustre: Skipped 3 previous similar messages [ 8955.430048] Lustre: Skipped 1 previous similar message [ 8961.004739] Lustre: server umount lustre-OST0000 complete [ 8970.727190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8972.303613] Lustre: server umount lustre-MDT0000 complete [ 8975.841863] LustreError: 324923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8975.875743] LustreError: 324923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8977.168107] LustreError: 321289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636057 with bad export cookie 18074220983157589119 [ 8977.188499] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8977.191555] LustreError: 321289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8977.717448] Lustre: server umount lustre-MDT0001 complete [ 8990.549150] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 8993.249226] Key type lgssc unregistered [ 8993.590905] LNet: 327596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8993.599289] LNetError: 327596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8994.661808] LNet: Removed LNI 192.168.203.124@tcp [ 8995.600810] Key type .llcrypt unregistered [ 8995.606480] Key type ._llcrypt unregistered [ 9020.144492] Key type ._llcrypt registered [ 9020.151034] Key type .llcrypt registered [ 9020.270137] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 9021.628838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9021.653533] alg: No test for adler32 (adler32-zlib) [ 9022.671529] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 9022.923153] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 9024.583234] Key type lgssc registered [ 9025.786747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9034.264444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9034.284898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9035.721166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9040.192163] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9045.984523] LustreError: 329010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9046.007331] LustreError: 329010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9048.303062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9048.594490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9052.768060] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9059.815471] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9065.457599] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9069.024418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9069.031980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9069.054916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9071.075889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9071.076708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9074.209847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9074.219880] Lustre: Skipped 2 previous similar messages [ 9074.726659] Lustre: server umount lustre-MDT0000 complete [ 9078.587471] LustreError: 328990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636159 with bad export cookie 3212821864294284662 [ 9078.603321] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9079.193330] Lustre: server umount lustre-MDT0001 complete [ 9093.955542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9094.317986] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9094.343980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9094.638507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9099.246324] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9107.914864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9108.041321] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9108.106692] Lustre: 331596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9108.449057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9113.502481] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9121.393487] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9127.804435] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9138.021881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9138.226562] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9138.245506] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9138.253590] LustreError: 333484:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9138.267783] LustreError: 333484:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9138.275667] LustreError: 333484:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9138.383756] Lustre: server umount lustre-OST0000 complete [ 9138.387613] LustreError: 333484:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9148.209734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9148.454196] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9148.704457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9154.298505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9155.317544] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9163.028855] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9170.493346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9170.575712] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9170.664962] Lustre: lustre-OST0001: new disk, initializing [ 9170.676722] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9170.681031] Lustre: 335023:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9170.790570] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9177.411890] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9178.231855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9178.241734] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9178.354443] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9184.627254] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9188.832461] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9188.846422] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9188.853904] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9188.880826] Lustre: Skipped 1 previous similar message [ 9193.971212] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9193.974789] Lustre: Skipped 2 previous similar messages [ 9202.143291] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9202.327412] Lustre: server umount lustre-OST0001 complete [ 9204.204531] LustreError: 335644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9204.240746] LustreError: 335644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9209.314467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9209.316697] LustreError: 333952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9209.319650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9209.319658] Lustre: Skipped 2 previous similar messages [ 9209.320281] Lustre: Skipped 1 previous similar message [ 9209.332917] LustreError: 333952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9212.417977] Lustre: server umount lustre-OST0000 complete [ 9219.554193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9219.567452] Lustre: Skipped 1 previous similar message [ 9229.792336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9229.809247] Lustre: Skipped 6 previous similar messages [ 9230.303254] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9230.742065] Lustre: server umount lustre-MDT0000 complete [ 9234.748932] LustreError: 332346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636315 with bad export cookie 3212821864294285215 [ 9234.767744] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9234.768655] LustreError: 332346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9235.184898] Lustre: server umount lustre-MDT0001 complete [ 9246.737965] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 9250.206542] Key type lgssc unregistered [ 9250.516835] LNet: 336987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9250.531785] LNetError: 336987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9250.554802] LNet: Removed LNI 192.168.203.124@tcp [ 9251.581190] Key type .llcrypt unregistered [ 9251.583725] Key type ._llcrypt unregistered [ 9326.879732] Key type ._llcrypt registered [ 9326.883571] Key type .llcrypt registered [ 9327.016637] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 9342.248913] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 9343.557938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9343.581512] alg: No test for adler32 (adler32-zlib) [ 9344.724351] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 9345.091480] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 9346.791144] Key type lgssc registered [ 9348.050651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9379.163990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9379.188128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9380.446443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9380.478656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9380.580779] Lustre: lustre-MDT0000: new disk, initializing [ 9380.649866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9380.668189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9385.197235] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9394.698933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9394.839919] Lustre: 341935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9394.902918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9394.906869] Lustre: Skipped 1 previous similar message [ 9394.981574] Lustre: lustre-MDT0001: new disk, initializing [ 9395.096725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9395.137139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9395.157583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9399.594926] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9408.103758] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9413.357593] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9419.257900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9419.548325] Lustre: lustre-OST0000: new disk, initializing [ 9419.559318] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9419.565270] Lustre: 343903:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9419.677929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9425.087201] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9429.591284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9429.608619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9429.748533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9433.351899] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9437.537181] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9437.722071] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9441.740916] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9442.117540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9445.346037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9445.350972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9445.379851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9445.386932] Lustre: Skipped 1 previous similar message [ 9450.403089] Lustre: server umount lustre-OST0000 complete [ 9455.585200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9455.589795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9455.605807] Lustre: Skipped 2 previous similar messages [ 9460.704478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9460.715146] Lustre: Skipped 2 previous similar messages [ 9465.825574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9465.833384] Lustre: Skipped 1 previous similar message [ 9468.383119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9468.896882] Lustre: server umount lustre-MDT0000 complete [ 9472.806236] LustreError: 341927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636553 with bad export cookie 7507703217595140692 [ 9472.810268] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9472.822234] LustreError: 341927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9473.176600] Lustre: server umount lustre-MDT0001 complete [ 9482.633185] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:56:00 (1786636560) [ 9492.235730] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:56:10 (1786636570) [ 9502.380882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9503.055553] LustreError: 346664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9503.067592] LustreError: 346664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9503.168115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9507.195606] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9508.322625] LustreError: 346665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9512.417311] LustreError: 346664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9515.599978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9516.236423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9520.496165] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9528.669663] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9534.406283] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9541.056728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9541.418062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9546.978580] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9553.768256] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9565.379193] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9567.169934] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9568.661903] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9570.504591] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9572.743837] Lustre: Failing over lustre-MDT0000 [ 9573.437778] Lustre: server umount lustre-MDT0000 complete [ 9576.417839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9576.425107] LustreError: 346659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9576.451197] Lustre: Skipped 1 previous similar message [ 9576.472761] LustreError: 346659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9581.537709] LustreError: 346665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9581.557272] LustreError: 346665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9590.120177] LustreError: 346658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9590.161816] LustreError: 346658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9592.609572] Lustre: 340159:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786636657/real 1786636657] req@00000000373ae22b x1873424076049152/t0(0) o400->MGC192.168.203.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786636673 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9592.651105] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9593.229705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9603.042101] LustreError: 340156:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fb5cde2d x1873424076054272/t0(0) o250->MGC192.168.203.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9603.462798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9603.525832] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9608.556633] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9608.628757] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9608.729264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9608.839682] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9608.899743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9617.982194] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9619.510838] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9623.211951] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9625.320924] Lustre: Failing over lustre-MDT0000 [ 9625.636374] Lustre: server umount lustre-MDT0000 complete [ 9629.068282] LustreError: 346658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9629.111158] LustreError: 346658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 9629.154953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9629.169066] Lustre: Skipped 2 previous similar messages [ 9644.419342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9644.511082] Lustre: 340159:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786636709/real 1786636709] req@00000000b1d2df70 x1873424076070400/t0(0) o400->MGC192.168.203.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786636725 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9644.554474] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9655.046603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9655.111793] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9655.668431] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9659.430107] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9660.406150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9660.410160] Lustre: Skipped 2 previous similar messages [ 9660.440419] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9660.505974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9669.125937] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9670.830106] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9680.869742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9680.883601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9680.893787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9680.908240] Lustre: Skipped 2 previous similar messages [ 9680.932582] Lustre: Skipped 2 previous similar messages [ 9683.724919] Lustre: server umount lustre-MDT0000 complete [ 9685.989095] LustreError: 346665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9685.999037] LustreError: 346665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 9688.174985] LustreError: 346644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636768 with bad export cookie 7507703217595143436 [ 9688.187529] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9688.192649] LustreError: 346644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9688.538463] Lustre: server umount lustre-MDT0001 complete [ 9703.143794] Lustre: server umount lustre-OST0000 complete [ 9718.007054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9722.865562] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9731.652375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9731.997506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9732.009915] Lustre: Skipped 1 previous similar message [ 9737.003847] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9744.103673] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9749.757922] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9757.059834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9764.598788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9765.907987] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9773.494810] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9785.312678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9785.321297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9785.330745] Lustre: Skipped 1 previous similar message [ 9785.337969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9787.364940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9789.883410] Lustre: server umount lustre-OST0000 complete [ 9800.476245] Lustre: server umount lustre-MDT0000 complete [ 9802.720881] LustreError: 353031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9802.744172] LustreError: 353031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9804.227537] LustreError: 353011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636884 with bad export cookie 7507703217595144157 [ 9804.234375] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9804.255869] LustreError: 353011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9804.800272] Lustre: server umount lustre-MDT0001 complete [ 9815.437260] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 9818.057775] Key type lgssc unregistered [ 9818.320289] LNet: 356578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9818.333824] LNetError: 356578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9818.356562] LNet: Removed LNI 192.168.203.124@tcp [ 9818.980870] Key type .llcrypt unregistered [ 9818.983170] Key type ._llcrypt unregistered [ 9841.683384] Key type ._llcrypt registered [ 9841.684729] Key type .llcrypt registered [ 9841.817966] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 9842.885263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9842.912183] alg: No test for adler32 (adler32-zlib) [ 9843.976571] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 9844.274702] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 9846.007210] Key type lgssc registered [ 9847.067899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9854.118379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9854.138644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9855.772329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9860.557520] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9866.218771] LustreError: 358091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9866.232646] LustreError: 358091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9869.054641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9869.460085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9873.810582] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9880.846140] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9885.757212] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9892.057805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9892.465767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9899.027467] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9900.655396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9906.496566] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9916.385417] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9916.391527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9916.396360] Lustre: Skipped 1 previous similar message [ 9921.511348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9921.516395] Lustre: Skipped 2 previous similar messages [ 9922.617655] Lustre: server umount lustre-MDT0000 complete [ 9925.564500] LustreError: 358071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637006 with bad export cookie 5886385969351163382 [ 9925.566483] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9925.574818] LustreError: 358071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9925.972619] Lustre: server umount lustre-MDT0001 complete [ 9938.574238] Lustre: server umount lustre-OST0000 complete [ 9952.846443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9953.303562] LustreError: 361803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9953.396742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9957.369077] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9958.890543] LustreError: 361804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9964.001898] LustreError: 361803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9965.783826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9966.157466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9970.956680] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9978.620526] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9984.403822] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9990.961365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9991.314765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9997.451520] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9999.545409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10005.600620] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10021.347036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10021.354820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10021.369861] Lustre: Skipped 3 previous similar messages [10021.380554] Lustre: Skipped 2 previous similar messages [10023.918658] Lustre: server umount lustre-MDT0000 complete [10026.467288] LustreError: 363538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10026.484164] LustreError: 363538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10027.470444] LustreError: 361786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637108 with bad export cookie 5886385969351164495 [10027.483176] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10027.485830] LustreError: 361786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10027.836275] Lustre: server umount lustre-MDT0001 complete [10041.934365] Lustre: server umount lustre-OST0000 complete [10056.059897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10056.441445] LustreError: 365712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10056.498874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10061.187350] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10068.435136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10072.370704] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10079.577484] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10085.731079] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10092.027285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10092.379787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10092.384842] Lustre: Skipped 1 previous similar message [10098.774891] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10101.618605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10106.628922] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10114.025714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10114.033756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10114.050270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10117.089252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10117.099113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10119.197639] Lustre: server umount lustre-OST0000 complete [10125.281940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10125.298927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10125.328177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10125.339021] Lustre: Skipped 2 previous similar messages [10129.175387] Lustre: server umount lustre-MDT0000 complete [10132.448143] LustreError: 366787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10132.477241] LustreError: 366787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10132.647366] LustreError: 365693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637213 with bad export cookie 5886385969351165517 [10132.649654] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10132.653211] LustreError: 365693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10132.932334] Lustre: server umount lustre-MDT0001 complete [10142.580769] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [10145.200886] Key type lgssc unregistered [10145.529440] LNet: 369247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10145.534220] LNetError: 369247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10145.553656] LNet: Removed LNI 192.168.203.124@tcp [10146.303642] Key type .llcrypt unregistered [10146.306840] Key type ._llcrypt unregistered [10169.006509] Key type ._llcrypt registered [10169.009937] Key type .llcrypt registered [10169.134346] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [10170.372473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10170.392817] alg: No test for adler32 (adler32-zlib) [10171.524553] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [10171.773879] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [10173.423231] Key type lgssc registered [10174.510981] Lustre: Echo OBD driver; http://www.lustre.org/ [10181.987788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10182.039410] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10182.051842] LustreError: 370688:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10182.084673] LustreError: 370688:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10182.107634] LustreError: 370688:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10182.124242] LustreError: 370688:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10182.137377] LustreError: 370688:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10189.764234] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10189.770993] LustreError: 371086:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10189.787491] LustreError: 371086:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10189.792692] LustreError: 371086:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10189.797490] LustreError: 371086:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10189.803844] LustreError: 371086:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10209.381516] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [10212.607981] Key type lgssc unregistered [10212.974505] LNet: 372092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10212.997754] LNetError: 372092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10213.018560] LNet: Removed LNI 192.168.203.124@tcp [10214.138172] Key type .llcrypt unregistered [10214.143666] Key type ._llcrypt unregistered [10232.190765] Key type ._llcrypt registered [10232.193535] Key type .llcrypt registered [10232.277647] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [10247.554738] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [10248.511114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10248.540734] alg: No test for adler32 (adler32-zlib) [10249.760093] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [10250.262952] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [10251.927269] Key type lgssc registered [10253.468728] Lustre: Echo OBD driver; http://www.lustre.org/ [10285.929731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10285.969304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10287.200463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10287.232124] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10287.323136] Lustre: lustre-MDT0000: new disk, initializing [10287.407133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10287.434563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10291.233171] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10300.150857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10300.274771] Lustre: 375249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10300.323972] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10300.328350] Lustre: Skipped 1 previous similar message [10300.400743] Lustre: lustre-MDT0001: new disk, initializing [10300.476503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10300.516884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10300.532842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10304.528190] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10313.004406] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10318.158140] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10324.662631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10324.852834] Lustre: lustre-OST0000: new disk, initializing [10324.856608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10324.862822] Lustre: 377218:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10324.960147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10330.725887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10330.730343] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10330.736676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10330.834661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10340.923838] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10345.757542] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10346.079416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10351.169727] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10351.388498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10356.708089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10356.711308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10356.724431] LustreError: Skipped 1 previous similar message [10356.732246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10359.716432] Lustre: server umount lustre-OST0000 complete [10366.946555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10366.948173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10366.976160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10366.981727] Lustre: Skipped 2 previous similar messages [10366.993502] Lustre: Skipped 2 previous similar messages [10369.211725] Lustre: server umount lustre-MDT0000 complete [10372.202780] LustreError: 375239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637452 with bad export cookie 10992879674995947368 [10372.214517] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10372.219380] LustreError: 375239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10372.580351] Lustre: server umount lustre-MDT0001 complete [10379.613534] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10381.177204] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:11:00 (1786637460) [10387.713398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10388.165881] LustreError: 379678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.189258] LustreError: 379678:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10388.232412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10392.402107] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10393.573970] LustreError: 379679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10397.670938] LustreError: 379678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10400.001832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10400.339933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10404.774678] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10411.475891] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10417.143818] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10423.848061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10424.252179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10430.283443] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10436.614415] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10442.412625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10442.612928] Lustre: lustre-OST0001: new disk, initializing [10442.620884] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10442.634348] Lustre: 382513:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10442.726587] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10449.891971] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10452.086609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10452.098156] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10452.176098] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10460.055493] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10467.183797] Lustre: lustre-OST0001: Not available for connect from 192.168.203.24@tcp (stopping) [10467.816073] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10467.816768] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10467.832163] LustreError: Skipped 1 previous similar message [10469.729668] Lustre: server umount lustre-OST0001 complete [10472.290055] LustreError: 381498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10478.057995] LustreError: 381499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10478.077576] LustreError: 381499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10486.755335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10486.761678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10486.767777] Lustre: Skipped 1 previous similar message [10486.771029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10486.774450] Lustre: Skipped 2 previous similar messages [10488.305916] LustreError: 382979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10488.337520] LustreError: 382979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10491.163319] Lustre: server umount lustre-OST0000 complete [10497.504652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10497.516913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10497.536956] Lustre: Skipped 1 previous similar message [10497.544895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10497.551941] Lustre: Skipped 2 previous similar messages [10500.871570] Lustre: server umount lustre-MDT0000 complete [10503.916016] LustreError: 379659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637584 with bad export cookie 10992879674995948250 [10503.920707] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10503.929694] LustreError: 379659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10510.329405] Lustre: server umount lustre-MDT0001 complete [10520.303588] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [10522.928876] Key type lgssc unregistered [10523.208628] LNet: 384505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10523.220535] LNetError: 384505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10524.261702] LNet: Removed LNI 192.168.203.124@tcp [10525.056732] Key type .llcrypt unregistered [10525.060517] Key type ._llcrypt unregistered [10572.456297] Key type ._llcrypt registered [10572.458176] Key type .llcrypt registered [10572.552861] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [10573.501246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10573.562620] alg: No test for adler32 (adler32-zlib) [10574.554861] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [10574.791942] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [10576.503323] Key type lgssc registered [10577.571888] Lustre: Echo OBD driver; http://www.lustre.org/ [10586.905816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10586.923629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10588.276711] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10588.300437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10588.495967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10594.035153] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10602.508978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10602.677210] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10602.706843] Lustre: 387306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10603.024146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10606.951517] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10612.693265] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10618.471366] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10622.948912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10622.960564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10622.971017] Lustre: Skipped 1 previous similar message [10622.987425] Lustre: Skipped 1 previous similar message [10626.685748] Lustre: server umount lustre-MDT0000 complete [10630.477394] LustreError: 387295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637711 with bad export cookie 6542174696543804342 [10630.481383] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10630.488658] LustreError: 387295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10630.908779] Lustre: server umount lustre-MDT0001 complete [10639.597685] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:15:18 (1786637718) [10646.059276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10646.416419] LustreError: 389815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10646.443467] LustreError: 389815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10646.487064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10651.014448] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10651.619846] LustreError: 389816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10656.743620] LustreError: 389815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10659.335319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10659.746629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10665.418027] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10674.142648] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10680.440628] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10688.013091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10688.250096] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10688.474714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10695.395920] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10703.402617] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10710.752785] Lustre: Permanently deactivating lustre-OST0000 [10710.767477] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10717.706806] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10719.395352] LustreError: 392801:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10719.406150] LustreError: 392801:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10719.410390] LustreError: 392801:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10721.758937] LustreError: 392849:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10721.763767] LustreError: 392849:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10729.656548] Lustre: server umount lustre-OST0000 complete [10733.190716] LustreError: 393099:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10733.198902] LustreError: 393099:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10736.610145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10736.627446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10736.648952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10739.683280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10739.689891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10739.715652] Lustre: Skipped 1 previous similar message [10744.805981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10744.820567] Lustre: Skipped 1 previous similar message [10746.017703] Lustre: server umount lustre-MDT0000 complete [10749.924892] LustreError: 389811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10749.961863] LustreError: 389811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10750.026449] LustreError: 389797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637830 with bad export cookie 6542174696543804923 [10750.033721] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10750.049676] LustreError: 389797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10759.814904] Lustre: server umount lustre-MDT0001 complete [10768.477699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10772.939571] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10775.703307] LustreError: 394257:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10781.089337] Lustre: 394597:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.124@tcp [10783.105098] Lustre: server umount MGS complete [10797.042522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10797.531968] LustreError: 395323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10797.565430] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10797.571941] Lustre: Skipped 1 previous similar message [10797.638301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10802.420206] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10807.783388] LustreError: 395324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10807.805877] LustreError: 395324:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10811.431348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10811.709732] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10816.666699] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10824.072261] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10830.008369] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10831.728889] Lustre: Permanently reactivating lustre-OST0000 [10831.752070] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10831.757279] Lustre: Skipped 2 previous similar messages [10861.023505] LustreError: 397154:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10869.716558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10870.090572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10870.097479] Lustre: Skipped 1 previous similar message [10876.458886] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10884.200787] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10891.748895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10891.749652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10891.756120] LustreError: Skipped 1 previous similar message [10891.772178] Lustre: Skipped 1 previous similar message [10891.813458] LustreError: 398126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10891.891893] Lustre: server umount lustre-OST0000 complete [10896.871381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10896.883034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10896.895925] Lustre: Skipped 1 previous similar message [10901.792649] Lustre: server umount lustre-MDT0000 complete [10905.568429] LustreError: 395305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637986 with bad export cookie 6542174696543805966 [10905.584230] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10905.587825] LustreError: 395305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10905.917258] Lustre: server umount lustre-MDT0001 complete [10915.868839] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [10918.369746] Key type lgssc unregistered [10918.668925] LNet: 399834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10918.677582] LNetError: 399834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10918.710485] LNet: Removed LNI 192.168.203.124@tcp [10919.488083] Key type .llcrypt unregistered [10919.491104] Key type ._llcrypt unregistered [10937.083525] Key type ._llcrypt registered [10937.092170] Key type .llcrypt registered [10937.206661] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [10953.189549] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [10954.627148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10954.720271] alg: No test for adler32 (adler32-zlib) [10955.772296] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [10956.093109] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [10957.831156] Key type lgssc registered [10958.993553] Lustre: Echo OBD driver; http://www.lustre.org/ [10994.726705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10994.764227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10996.047407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10996.106410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10996.199966] Lustre: lustre-MDT0000: new disk, initializing [10996.323781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10996.365255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11000.786101] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11012.240812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11012.468728] Lustre: 403005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11012.584913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11012.605259] Lustre: Skipped 1 previous similar message [11012.747767] Lustre: lustre-MDT0001: new disk, initializing [11012.877822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11012.924863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11012.944966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11018.129402] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11028.128433] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11034.055586] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11040.065335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11040.408383] Lustre: lustre-OST0000: new disk, initializing [11040.411939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11040.421146] Lustre: 404973:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11040.510080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11046.182036] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11047.491551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11047.513029] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11047.618685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11055.421867] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11060.201597] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11060.444380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11064.746250] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11064.997065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11068.388926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11068.389049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11068.395699] LustreError: Skipped 1 previous similar message [11068.402072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11068.411986] Lustre: Skipped 1 previous similar message [11068.428462] Lustre: Skipped 1 previous similar message [11073.504260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11078.645173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11078.661735] Lustre: Skipped 2 previous similar messages [11081.183196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11081.284187] Lustre: server umount lustre-OST0000 complete [11084.779723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11084.791626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11084.808248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11088.870082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11088.881428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11088.897683] Lustre: Skipped 1 previous similar message [11091.080069] Lustre: server umount lustre-MDT0000 complete [11094.863745] LustreError: 402995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638175 with bad export cookie 14155038741537676598 [11094.864936] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11094.876647] LustreError: 402995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11095.246183] Lustre: server umount lustre-MDT0001 complete [11103.396435] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11104.917294] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:23:04 (1786638184) [11111.206620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11111.661047] LustreError: 407433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11111.688987] LustreError: 407433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11111.759246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11116.322803] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11117.035586] LustreError: 407434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11122.151292] LustreError: 407433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11124.701677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11124.946575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11129.377547] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11136.789813] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11143.696391] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11150.495171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11150.829936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11156.881853] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11163.422996] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11165.204976] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11167.672619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11177.441338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11177.449093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11177.460465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11180.514217] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11183.539185] Lustre: server umount lustre-OST0000 complete [11190.761272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11190.790290] Lustre: Skipped 2 previous similar messages [11193.696483] Lustre: server umount lustre-MDT0000 complete [11195.872743] LustreError: 407433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11195.889336] LustreError: 407433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11197.175193] LustreError: 407415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638277 with bad export cookie 14155038741537677487 [11197.192319] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11197.509568] Lustre: server umount lustre-MDT0001 complete [11207.698878] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [11210.663668] Key type lgssc unregistered [11210.906486] LNet: 411164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11210.911450] LNetError: 411164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11210.919711] LNet: Removed LNI 192.168.203.124@tcp [11211.864134] Key type .llcrypt unregistered [11211.866204] Key type ._llcrypt unregistered [11236.273833] Key type ._llcrypt registered [11236.277594] Key type .llcrypt registered [11236.380165] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [11236.701428] Key type .llcrypt unregistered [11236.705432] Key type ._llcrypt unregistered [11254.596947] Key type ._llcrypt registered [11254.598670] Key type .llcrypt registered [11254.711642] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [11255.875749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11255.920212] alg: No test for adler32 (adler32-zlib) [11257.045882] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [11257.285544] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [11259.039154] Key type lgssc registered [11260.388692] Lustre: Echo OBD driver; http://www.lustre.org/ [11269.723844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11269.761990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11271.287125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11276.161449] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11280.869187] LustreError: 413382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11280.893801] LustreError: 413382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11285.388502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11285.701239] LustreError: 413381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11285.731217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11285.731977] LustreError: 413381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11291.112549] LustreError: 413382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11291.143199] LustreError: 413382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11292.332297] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11296.228613] LustreError: 413381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11296.256517] LustreError: 413381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11296.282445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11300.337242] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11301.352966] LustreError: 414126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11301.371154] LustreError: 414126:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11307.620582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11308.101845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11312.711663] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11321.829426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11321.834628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11321.866202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11323.362347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11323.379081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11323.385742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11327.655607] Lustre: server umount lustre-OST0000 complete [11333.602358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11333.608973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11333.610547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11333.610553] Lustre: Skipped 1 previous similar message [11333.619801] Lustre: Skipped 1 previous similar message [11337.808819] Lustre: server umount lustre-MDT0000 complete [11338.730160] LustreError: 413377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.746330] LustreError: 413377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11341.012468] LustreError: 413361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638421 with bad export cookie 11011965490597905710 [11341.017073] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11341.028541] LustreError: 413361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11341.433089] Lustre: server umount lustre-MDT0001 complete [11353.484197] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [11356.159683] Key type lgssc unregistered [11356.446429] LNet: 416532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11356.463550] LNetError: 416532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11357.548450] LNet: Removed LNI 192.168.203.124@tcp [11358.489161] Key type .llcrypt unregistered [11358.492263] Key type ._llcrypt unregistered [11384.156615] Key type ._llcrypt registered [11384.168629] Key type .llcrypt registered [11384.342559] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [11385.396510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11385.451627] alg: No test for adler32 (adler32-zlib) [11386.896258] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [11387.295291] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [11389.111245] Key type lgssc registered [11390.398794] Lustre: Echo OBD driver; http://www.lustre.org/ [11401.208392] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11401.224699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11416.863438] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11421.984057] LustreError: 417925:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.124@tcp: failed processing log, type 4: rc = -110 [11448.671285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11455.122559] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11462.314196] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11468.793348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11468.927138] LustreError: 417259:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000677a3cf2 x1873426217175296/t0(0) o250->MGC192.168.203.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11469.288903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11469.350060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11473.729608] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11474.403168] LustreError: 417951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11474.417727] LustreError: 417951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11479.522549] LustreError: 417950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11481.657812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11482.031831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11486.133660] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11493.250817] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11498.736930] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11505.120231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11505.126591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11505.149448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11507.073044] Lustre: server umount lustre-OST0000 complete [11512.802195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11512.816331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11512.818960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11512.826422] LustreError: Skipped 1 previous similar message [11514.849735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11514.851508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11516.563940] Lustre: server umount lustre-MDT0000 complete [11519.975572] LustreError: 418962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11519.998184] LustreError: 418962:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11520.838644] LustreError: 417932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638601 with bad export cookie 216786196164032287 [11520.845400] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11520.862441] LustreError: 417932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11531.343155] Lustre: server umount lustre-MDT0001 complete [11542.578712] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [11545.506310] Key type lgssc unregistered [11545.871055] LNet: 421499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11545.886159] LNetError: 421499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11545.902894] LNet: Removed LNI 192.168.203.124@tcp [11546.876469] Key type .llcrypt unregistered [11546.883588] Key type ._llcrypt unregistered [11570.135922] Key type ._llcrypt registered [11570.137295] Key type .llcrypt registered [11570.216241] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [11571.165843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11571.198967] alg: No test for adler32 (adler32-zlib) [11572.263638] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [11572.526828] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [11574.231884] Key type lgssc registered [11575.413944] Lustre: Echo OBD driver; http://www.lustre.org/ [11582.718734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11582.767694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11584.240325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11588.472562] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11593.699381] LustreError: 422916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.717120] LustreError: 422916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11596.204465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11596.557364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11600.950733] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11607.815456] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11612.711850] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11618.408708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11618.816076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11623.460231] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11629.064784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11629.498832] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11634.155482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11634.170878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11634.175377] Lustre: Skipped 1 previous similar message [11634.192389] Lustre: Skipped 2 previous similar messages [11636.115729] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [11639.266644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11639.269635] Lustre: Skipped 2 previous similar messages [11641.199369] LustreError: 424156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11641.883715] Lustre: server umount lustre-MDT0000 complete [11644.126744] LustreError: 422894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.24@tcp arrived at 1786638724 with bad export cookie 7029471567875609619 [11644.387534] LustreError: 422916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11644.419634] LustreError: 422916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11645.865253] LustreError: 422896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638726 with bad export cookie 7029471567875608723 [11645.875692] LustreError: 422896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11645.882480] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11647.461027] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11647.465569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11647.479141] Lustre: Skipped 1 previous similar message [11647.484177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11649.504089] LustreError: 423643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11649.516352] LustreError: 423643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11652.159208] Lustre: server umount lustre-OST0000 complete [11655.648777] LustreError: 423643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11655.659647] LustreError: 423643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11658.643474] Lustre: server umount lustre-MDT0001 complete [11668.430334] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [11670.864342] Key type lgssc unregistered [11671.126544] LNet: 426602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11671.132760] LNetError: 426602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11671.149626] LNet: Removed LNI 192.168.203.124@tcp [11671.808319] Key type .llcrypt unregistered [11671.810722] Key type ._llcrypt unregistered [11693.133583] Key type ._llcrypt registered [11693.137295] Key type .llcrypt registered [11693.260818] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [11694.742142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11694.761074] alg: No test for adler32 (adler32-zlib) [11695.747511] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [11695.988749] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [11697.671508] Key type lgssc registered [11698.760820] Lustre: Echo OBD driver; http://www.lustre.org/ [11705.219491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11705.256844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11706.644712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11710.972588] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11717.089675] LustreError: 428018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11717.096807] LustreError: 428018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11719.512123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11720.000986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11724.733882] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11731.666874] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11736.479854] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11742.273422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11742.523924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11748.036636] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11751.748389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11755.049557] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11761.123801] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11761.131341] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11761.141337] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11761.399323] Lustre: server umount lustre-MDT0001 complete [11762.069370] LustreError: 428014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11762.156366] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11767.266036] LustreError: 428015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11767.294407] LustreError: 428015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11772.385579] LustreError: 428018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11772.390144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11772.421854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11772.440115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11773.857597] Lustre: server umount lustre-OST0000 complete [11777.565723] Lustre: server umount lustre-MDT0000 complete [11791.111570] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [11794.079171] Key type lgssc unregistered [11794.393500] LNet: 431705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11794.407123] LNetError: 431705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11795.431070] LNet: Removed LNI 192.168.203.124@tcp [11796.334264] Key type .llcrypt unregistered [11796.338668] Key type ._llcrypt unregistered [11818.833923] Key type ._llcrypt registered [11818.836697] Key type .llcrypt registered [11818.988883] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [11833.255589] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [11834.426331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11834.466180] alg: No test for adler32 (adler32-zlib) [11835.796107] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [11836.121735] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [11837.800493] Key type lgssc registered [11839.522517] Lustre: Echo OBD driver; http://www.lustre.org/ [11880.071728] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [11895.907359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11895.938791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11897.263713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11897.296873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11897.384662] Lustre: lustre-MDT0000: new disk, initializing [11897.524314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11897.542959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11901.786226] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11915.104111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11915.311643] Lustre: lustre-OST0000: new disk, initializing [11915.315102] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11915.319338] Lustre: Skipped 1 previous similar message [11915.328912] Lustre: 436739:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11915.407560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11921.416291] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11923.586735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11923.594924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11923.623588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11930.629681] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11945.407271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11945.523080] Lustre: 435722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11945.585980] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11945.643927] Lustre: lustre-MDT0001: new disk, initializing [11945.783314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11945.842635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11945.857294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11950.364205] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11959.020290] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11959.215252] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11963.435510] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11963.632762] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11970.018701] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11970.022553] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11970.034179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11970.051457] Lustre: Skipped 1 previous similar message [11971.957640] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.24@tcp (stopping) [11975.146892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11980.263176] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11980.274854] Lustre: Skipped 3 previous similar messages [11982.304724] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11982.418746] Lustre: server umount lustre-MDT0001 complete [11990.505213] LustreError: 435734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11990.552135] LustreError: 435734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11995.616361] LustreError: 436747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11995.629559] LustreError: 436747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12000.740960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12000.746228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12000.757311] Lustre: Skipped 1 previous similar message [12000.766753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12000.775745] LustreError: 435733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12000.779122] Lustre: Skipped 1 previous similar message [12005.859814] LustreError: 435734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12010.979430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12010.980168] LustreError: 435733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12010.985470] Lustre: Skipped 1 previous similar message [12014.048217] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12014.340811] Lustre: server umount lustre-OST0000 complete [12024.856300] Lustre: server umount lustre-MDT0000 complete [12038.536475] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [12041.312939] Key type lgssc unregistered [12041.671975] LNet: 440493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12041.677451] LNetError: 440493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12041.690834] LNet: Removed LNI 192.168.203.124@tcp [12042.586108] Key type .llcrypt unregistered [12042.589687] Key type ._llcrypt unregistered [12078.925634] Key type ._llcrypt registered [12078.928240] Key type .llcrypt registered [12079.005397] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [12079.283318] Key type .llcrypt unregistered [12079.286236] Key type ._llcrypt unregistered [12094.629250] Key type ._llcrypt registered [12094.632513] Key type .llcrypt registered [12094.723160] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [12095.635315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12095.741244] alg: No test for adler32 (adler32-zlib) [12096.893952] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [12097.201603] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [12098.895200] Key type lgssc registered [12099.862927] Lustre: Echo OBD driver; http://www.lustre.org/ [12123.364147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12123.399971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12124.600845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12124.643995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12124.787963] Lustre: lustre-MDT0000: new disk, initializing [12124.899138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12124.915147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12129.108172] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [12138.493205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12138.630462] Lustre: 443875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12138.676475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12138.679671] Lustre: Skipped 1 previous similar message [12138.731712] Lustre: lustre-MDT0001: new disk, initializing [12138.802670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12138.833720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12138.846967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12143.323508] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [12151.880564] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12156.970819] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12162.367305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12162.606697] Lustre: lustre-OST0000: new disk, initializing [12162.609856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12162.615264] Lustre: 445845:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12162.706820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12168.251942] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [12169.352762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12169.366626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12169.463844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12177.134513] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12185.058912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12185.073955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12185.099479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12185.120254] Lustre: Skipped 1 previous similar message [12190.342673] Lustre: server umount lustre-MDT0000 complete [12194.228956] LustreError: 443867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786639274 with bad export cookie 13810391243859678850 [12194.237842] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12194.248638] LustreError: 443867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12194.784726] Lustre: server umount lustre-MDT0001 complete [12209.021589] Lustre: server umount lustre-OST0000 complete [12219.386845] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:41:38 (1786639298) [12226.272714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12226.778636] LustreError: 447875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12226.795180] LustreError: 447875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12226.856432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12229.938545] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [12232.161642] LustreError: 447876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12237.282446] LustreError: 447875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12238.493300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12238.911843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12243.609068] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [12251.531995] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12257.833401] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12278.824314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12279.043497] Lustre: Found index 0 for lustre-OST0000, updating log [12279.321383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12285.561884] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [12294.457896] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12297.877477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12308.450236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12308.454971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12308.462664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12308.466541] Lustre: Skipped 3 previous similar messages [12308.477495] Lustre: Skipped 1 previous similar message [12309.538603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12309.564921] Lustre: Skipped 1 previous similar message [12313.744619] Lustre: server umount lustre-OST0000 complete [12329.459445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12329.464070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12329.466558] Lustre: Skipped 1 previous similar message [12329.476110] Lustre: Skipped 1 previous similar message [12334.560523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12334.569346] Lustre: Skipped 1 previous similar message [12335.858045] Lustre: server umount lustre-MDT0000 complete [12336.611583] LustreError: 447875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12336.629323] LustreError: 447875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12339.695841] LustreError: 447856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786639420 with bad export cookie 13810391243859680243 [12339.698192] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12339.716148] LustreError: 447856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12340.010828] Lustre: server umount lustre-MDT0001 complete [12347.107158] LNet: 1 local NIs in recovery (showing 1): 192.168.203.124@tcp [12349.684529] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:43:48 (1786639428) [12973.671830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12974.215244] LustreError: 454308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12974.278889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12978.694437] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [12979.688336] LustreError: 454309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12983.784258] LustreError: 454308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12987.205933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12987.619252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12991.832548] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [12998.316449] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13003.353948] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13022.634110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13022.846621] Lustre: Found index 0 for lustre-OST0000, updating log [13023.128425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13026.022106] LNet: 1 local NIs in recovery (showing 1): 192.168.203.124@tcp [13029.058247] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [13036.596436] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13038.916154] Lustre: server umount lustre-OST0000 complete [13043.684508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13043.705529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13043.712511] Lustre: Skipped 2 previous similar messages [13044.773279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13044.782660] Lustre: Skipped 1 previous similar message [13046.822944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13046.829045] Lustre: Skipped 3 previous similar messages [13048.970362] Lustre: server umount lustre-MDT0000 complete [13049.893250] LustreError: 455362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13049.918965] LustreError: 455362:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13052.368578] LustreError: 456169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786640133 with bad export cookie 13810391243859681328 [13052.376904] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13052.385512] LustreError: 456169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13052.688334] Lustre: server umount lustre-MDT0001 complete [13083.838375] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [13085.410180] LNet: 1 local NIs in recovery (showing 1): 192.168.203.124@tcp [13094.932963] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [13131.980536] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:56:50 (1786640210) [13151.382962] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [13154.941659] Key type lgssc unregistered [13155.371902] LNet: 460932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13155.393410] LNetError: 460932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13284.454660] LNet: Removed LNI 192.168.203.124@tcp [13285.328316] Key type .llcrypt unregistered [13285.333106] Key type ._llcrypt unregistered [13302.003513] Key type ._llcrypt registered [13302.008918] Key type .llcrypt registered [13302.134494] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [13303.254123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13303.279600] alg: No test for adler32 (adler32-zlib) [13304.572131] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [13304.789963] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [13306.463548] Key type lgssc registered [13307.938984] Lustre: Echo OBD driver; http://www.lustre.org/ [13312.484657] LNet: Added LNI 192.168.203.124@tcp100 [8/256/0/180] [13313.495332] LNet: Added LNI 192.168.203.124@tcp101 [8/256/0/180] [13314.499146] LNet: Added LNI 192.168.203.124@tcp102 [8/256/0/180] [13315.420677] LNet: Added LNI 192.168.203.124@tcp103 [8/256/0/180] [13316.415281] LNet: Added LNI 192.168.203.124@tcp104 [8/256/0/180] [13317.233219] LNet: Added LNI 192.168.203.124@tcp105 [8/256/0/180] [13318.125931] LNet: Added LNI 192.168.203.124@tcp106 [8/256/0/180] [13319.186225] LNet: Added LNI 192.168.203.124@tcp107 [8/256/0/180] [13320.299075] LNet: Added LNI 192.168.203.124@tcp108 [8/256/0/180] [13321.217820] LNet: Added LNI 192.168.203.124@tcp109 [8/256/0/180] [13322.100689] LNet: Added LNI 192.168.203.124@tcp110 [8/256/0/180] [13323.044897] LNet: Added LNI 192.168.203.124@tcp111 [8/256/0/180] [13323.936975] LNet: Added LNI 192.168.203.124@tcp112 [8/256/0/180] [13324.812652] LNet: Added LNI 192.168.203.124@tcp113 [8/256/0/180] [13325.553193] LNet: Added LNI 192.168.203.124@tcp114 [8/256/0/180] [13326.480977] LNet: Added LNI 192.168.203.124@tcp115 [8/256/0/180] [13327.369494] LNet: Added LNI 192.168.203.124@tcp116 [8/256/0/180] [13328.414111] LNet: Added LNI 192.168.203.124@tcp117 [8/256/0/180] [13329.418574] LNet: Added LNI 192.168.203.124@tcp118 [8/256/0/180] [13330.392031] LNet: Added LNI 192.168.203.124@tcp119 [8/256/0/180] [13331.271216] LNet: Added LNI 192.168.203.124@tcp120 [8/256/0/180] [13332.102751] LNet: Added LNI 192.168.203.124@tcp121 [8/256/0/180] [13333.082914] LNet: Added LNI 192.168.203.124@tcp122 [8/256/0/180] [13334.039532] LNet: Added LNI 192.168.203.124@tcp123 [8/256/0/180] [13335.134895] LNet: Added LNI 192.168.203.124@tcp124 [8/256/0/180] [13336.046387] LNet: Added LNI 192.168.203.124@tcp125 [8/256/0/180] [13337.176777] LNet: Added LNI 192.168.203.124@tcp126 [8/256/0/180] [13338.196099] LNet: Added LNI 192.168.203.124@tcp127 [8/256/0/180] [13339.123042] LNet: Added LNI 192.168.203.124@tcp128 [8/256/0/180] [13340.165523] LNet: Added LNI 192.168.203.124@tcp129 [8/256/0/180] [13341.219108] LNet: Added LNI 192.168.203.124@tcp130 [8/256/0/180] [13342.291603] LNet: Added LNI 192.168.203.124@tcp131 [8/256/0/180] [13343.270740] LNet: Added LNI 192.168.203.124@tcp132 [8/256/0/180] [13344.148270] LNet: Added LNI 192.168.203.124@tcp133 [8/256/0/180] [13345.121588] LNet: Added LNI 192.168.203.124@tcp134 [8/256/0/180] [13345.997540] LNet: Added LNI 192.168.203.124@tcp135 [8/256/0/180] [13353.596331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13353.618661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13354.779648] Lustre: 463965:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.124@tcp: can't add NID 192.168.203.124@tcp130: rc = -75 [13354.899602] LNetError: 463986:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.124 [13354.920394] LNetError: Protocol error connecting to 192.168.203.124@tcp120 at host 192.168.203.124:988: is it running a compatible version of Lustre? [13354.939305] LNetError: Skipped 5 previous similar messages [13354.975176] LNet: 7 local NIs in recovery (showing 7): 192.168.203.124@tcp124, 192.168.203.124@tcp125, 192.168.203.124@tcp126, 192.168.203.124@tcp128, 192.168.203.124@tcp127, 192.168.203.124@tcp120, 192.168.203.124@tcp129 [13355.070026] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13355.122129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13355.261871] Lustre: lustre-MDT0000: new disk, initializing [13355.385303] Lustre: 464023:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.124@tcp: can't add NID 192.168.203.124@tcp130: rc = -75 [13355.394102] Lustre: 464023:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13355.415598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13355.430435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13355.936460] LNetError: 461428:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.124 [13355.954354] LNetError: 461428:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [13355.980349] LNetError: Protocol error connecting to 192.168.203.124@tcp112 at host 192.168.203.124:988: is it running a compatible version of Lustre? [13355.986664] LNetError: Skipped 3 previous similar messages [13358.123729] LNetError: 461427:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.124 [13358.147630] LNetError: 461427:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [13358.155552] LNetError: Protocol error connecting to 192.168.203.124@tcp125 at host 192.168.203.124:988: is it running a compatible version of Lustre? [13358.163881] LNetError: Skipped 8 previous similar messages [13360.432202] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [13362.274809] LNetError: 463997:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.124 [13362.282812] LNetError: 463997:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [13362.294305] LNetError: Protocol error connecting to 192.168.203.124@tcp125 at host 192.168.203.124:988: is it running a compatible version of Lustre? [13370.464553] LNetError: 463986:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.124 [13370.475616] LNetError: Protocol error connecting to 192.168.203.124@tcp125 at host 192.168.203.124:988: is it running a compatible version of Lustre? [13371.309207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13371.407304] Lustre: 463984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13371.451696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13371.455708] Lustre: Skipped 1 previous similar message [13371.516668] Lustre: lustre-MDT0001: new disk, initializing [13371.547984] Lustre: 464784:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.124@tcp: can't add NID 192.168.203.124@tcp130: rc = -75 [13371.554696] Lustre: 464784:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13371.679859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13371.719135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13371.728480] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13375.987629] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [13379.169917] Lustre: 465160:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.124@tcp: can't add NID 192.168.203.124@tcp130: rc = -75 [13379.184537] Lustre: 465160:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13384.673796] LNetError: 463998:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.124 [13384.701560] LNetError: Protocol error connecting to 192.168.203.124@tcp117 at host 192.168.203.124:988: is it running a compatible version of Lustre? [13385.699701] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13391.629309] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13398.198418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13398.508366] Lustre: lustre-OST0000: new disk, initializing [13398.511431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13398.520609] Lustre: 465969:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13398.624514] Lustre: 465974:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.124@tcp: can't add NID 192.168.203.124@tcp130: rc = -75 [13398.631249] Lustre: 465974:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13398.691351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13401.061748] LNetError: 461428:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.124 [13401.074167] LNetError: 461428:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [13401.081703] LNetError: Protocol error connecting to 192.168.203.124@tcp123 at host 192.168.203.124:988: is it running a compatible version of Lustre? [13401.090983] LNetError: Skipped 11 previous similar messages [13403.786316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13403.803424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13403.990546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13404.863927] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [13414.496078] LNet: 1 local NIs in recovery (showing 1): 192.168.203.124@tcp125 [13414.938439] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13424.624887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13424.635071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13424.647064] LustreError: Skipped 1 previous similar message [13424.647972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13424.665822] Lustre: Skipped 1 previous similar message [13424.680226] Lustre: Skipped 1 previous similar message [13428.530590] Lustre: server umount lustre-OST0000 complete [13433.316413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13433.331855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13433.350083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13434.851479] LNetError: 461428:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.124 [13434.865318] LNetError: 461428:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 21 previous similar messages [13434.869961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13434.870391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13434.875834] LNetError: Protocol error connecting to 192.168.203.124@tcp122 at host 192.168.203.124:988: is it running a compatible version of Lustre? [13434.875843] LNetError: Skipped 21 previous similar messages [13434.916784] Lustre: Skipped 1 previous similar message [13438.396229] Lustre: server umount lustre-MDT0000 complete [13442.044561] LustreError: 463974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786640522 with bad export cookie 1040708217931719026 [13442.054219] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13442.058975] LustreError: 463974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13442.442592] Lustre: server umount lustre-MDT0001 complete [13453.056853] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [13455.259287] Key type lgssc unregistered [13455.655630] LNet: 467887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13455.665561] LNetError: 467887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13455.672577] LNet: Removed LNI 192.168.203.124@tcp [13456.735428] LNet: Removed LNI 192.168.203.124@tcp100 [13456.745430] LNetError: 467887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13456.758394] LNetError: 467887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13456.766342] LNet: Removed LNI 192.168.203.124@tcp101 [13457.823266] LNet: Removed LNI 192.168.203.124@tcp102 [13457.828476] LNetError: 467887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13457.841605] LNetError: 467887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13457.852324] LNet: Removed LNI 192.168.203.124@tcp103 [13458.911224] LNet: Removed LNI 192.168.203.124@tcp104 [13459.935511] LNet: Removed LNI 192.168.203.124@tcp105 [13459.952527] LNetError: 467887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13459.965138] LNetError: 467887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13461.023163] LNet: Removed LNI 192.168.203.124@tcp106 [13462.047262] LNet: Removed LNI 192.168.203.124@tcp107 [13463.071244] LNet: Removed LNI 192.168.203.124@tcp108 [13464.095591] LNet: Removed LNI 192.168.203.124@tcp109 [13464.102390] LNetError: 467887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13464.109685] LNetError: 467887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [13464.115853] LNet: Removed LNI 192.168.203.124@tcp110 [13465.186333] LNet: Removed LNI 192.168.203.124@tcp111 [13466.207184] LNet: Removed LNI 192.168.203.124@tcp112 [13467.231618] LNet: Removed LNI 192.168.203.124@tcp113 [13468.255173] LNet: Removed LNI 192.168.203.124@tcp114 [13469.279942] LNet: Removed LNI 192.168.203.124@tcp115 [13470.304526] LNet: Removed LNI 192.168.203.124@tcp116 [13471.327764] LNet: Removed LNI 192.168.203.124@tcp117 [13472.351873] LNet: Removed LNI 192.168.203.124@tcp118 [13472.355845] LNetError: 467887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13472.365667] LNetError: 467887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13472.383860] LNet: Removed LNI 192.168.203.124@tcp119 [13473.442657] LNet: Removed LNI 192.168.203.124@tcp120 [13474.463146] LNet: Removed LNI 192.168.203.124@tcp121 [13475.487812] LNet: Removed LNI 192.168.203.124@tcp122 [13476.511231] LNet: Removed LNI 192.168.203.124@tcp123 [13477.535196] LNet: Removed LNI 192.168.203.124@tcp124 [13477.541592] LNet: Removed LNI 192.168.203.124@tcp125 [13478.559246] LNet: Removed LNI 192.168.203.124@tcp126 [13479.583200] LNet: Removed LNI 192.168.203.124@tcp127 [13480.612224] LNet: Removed LNI 192.168.203.124@tcp128 [13481.631177] LNet: Removed LNI 192.168.203.124@tcp129 [13481.633389] LNet: Removed LNI 192.168.203.124@tcp130 [13481.637427] LNet: Removed LNI 192.168.203.124@tcp131 [13481.641449] LNet: Removed LNI 192.168.203.124@tcp132 [13481.644660] LNet: Removed LNI 192.168.203.124@tcp133 [13481.659549] LNet: Removed LNI 192.168.203.124@tcp134 [13481.670763] LNet: Removed LNI 192.168.203.124@tcp135 [13482.573700] Key type .llcrypt unregistered [13482.576890] Key type ._llcrypt unregistered [13504.983848] Key type ._llcrypt registered [13504.989356] Key type .llcrypt registered [13505.122786] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [13520.961793] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [13521.956523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13521.997076] alg: No test for adler32 (adler32-zlib) [13523.229206] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [13523.577056] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [13525.266307] Key type lgssc registered [13526.552549] Lustre: Echo OBD driver; http://www.lustre.org/ [13558.602778] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:03:57 (1786640637) [13574.868472] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [13577.827315] Key type lgssc unregistered [13578.146922] LNet: 471979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13578.166208] LNetError: 471979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13578.190574] LNet: Removed LNI 192.168.203.124@tcp [13579.180319] Key type .llcrypt unregistered [13579.183904] Key type ._llcrypt unregistered [13593.371630] Key type ._llcrypt registered [13593.373708] Key type .llcrypt registered [13593.496076] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [13594.595816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13594.628028] alg: No test for adler32 (adler32-zlib) [13595.701221] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [13595.960148] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [13597.768832] Key type lgssc registered [13598.788530] Lustre: Echo OBD driver; http://www.lustre.org/ [13607.335294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13607.375188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13608.608671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13608.639212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13608.694186] Lustre: lustre-MDT0000: new disk, initializing [13608.765883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13608.779913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13612.748551] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [13621.902309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13622.035900] Lustre: 473176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13622.130532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13622.138565] Lustre: Skipped 1 previous similar message [13622.259814] Lustre: lustre-MDT0001: new disk, initializing [13622.360052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13622.396086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13622.402562] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13626.140759] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [13634.023210] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13638.761685] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13644.878766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13645.250318] Lustre: lustre-OST0000: new disk, initializing [13645.261542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13645.276778] Lustre: 475143:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13645.415211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13651.609431] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [13654.589038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13654.598792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13654.679053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13659.980296] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13663.527286] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing lnet_if_list [13668.479710] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13668.536770] LNet: Added LNI 192.168.203.124@tcp10 [8/256/0/180] [13674.262292] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13674.287515] LNet: Added LNI 192.168.203.124@tcp11 [8/256/0/180] [13677.920578] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13677.967472] LNet: Added LNI 192.168.203.124@tcp12 [8/256/0/180] [13681.791964] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13681.828267] LNet: Added LNI 192.168.203.124@tcp13 [8/256/0/180] [13685.409706] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13685.426604] LNet: Added LNI 192.168.203.124@tcp14 [8/256/0/180] [13689.420505] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13689.456114] LNet: Added LNI 192.168.203.124@tcp15 [8/256/0/180] [13694.293516] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13694.353578] LNet: Added LNI 192.168.203.124@tcp16 [8/256/0/180] [13699.430888] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13699.476365] LNet: Added LNI 192.168.203.124@tcp17 [8/256/0/180] [13704.411400] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13704.454993] LNet: Added LNI 192.168.203.124@tcp18 [8/256/0/180] [13709.435965] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13709.473864] LNet: Added LNI 192.168.203.124@tcp19 [8/256/0/180] [13714.220087] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13714.267978] LNet: Added LNI 192.168.203.124@tcp20 [8/256/0/180] [13720.020823] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13720.065498] LNet: Added LNI 192.168.203.124@tcp21 [8/256/0/180] [13726.009744] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13726.044038] LNet: Added LNI 192.168.203.124@tcp22 [8/256/0/180] [13731.482577] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13731.527400] LNet: Added LNI 192.168.203.124@tcp23 [8/256/0/180] [13737.776579] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13737.802675] LNet: Added LNI 192.168.203.124@tcp24 [8/256/0/180] [13743.755451] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13743.794658] LNet: Added LNI 192.168.203.124@tcp25 [8/256/0/180] [13749.552724] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13749.589551] LNet: Added LNI 192.168.203.124@tcp26 [8/256/0/180] [13755.377462] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13755.408963] LNet: Added LNI 192.168.203.124@tcp27 [8/256/0/180] [13762.617667] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13762.657156] LNet: Added LNI 192.168.203.124@tcp28 [8/256/0/180] [13768.764886] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13768.806792] LNet: Added LNI 192.168.203.124@tcp29 [8/256/0/180] [13774.671579] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13774.719990] LNet: Added LNI 192.168.203.124@tcp30 [8/256/0/180] [13781.031311] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13781.060233] LNet: Added LNI 192.168.203.124@tcp31 [8/256/0/180] [13786.082340] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13786.126292] LNet: Added LNI 192.168.203.124@tcp32 [8/256/0/180] [13793.376626] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13793.428321] LNet: Added LNI 192.168.203.124@tcp33 [8/256/0/180] [13800.991658] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13801.022510] LNet: Added LNI 192.168.203.124@tcp34 [8/256/0/180] [13808.878959] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13808.925182] LNet: Added LNI 192.168.203.124@tcp35 [8/256/0/180] [13816.021954] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13816.054265] LNet: Added LNI 192.168.203.124@tcp36 [8/256/0/180] [13823.620984] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13823.667522] LNet: Added LNI 192.168.203.124@tcp37 [8/256/0/180] [13831.024447] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13831.087808] LNet: Added LNI 192.168.203.124@tcp38 [8/256/0/180] [13838.330883] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13838.363777] LNet: Added LNI 192.168.203.124@tcp39 [8/256/0/180] [13846.104402] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13846.129874] LNet: Added LNI 192.168.203.124@tcp40 [8/256/0/180] [13853.247963] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13853.306979] LNet: Added LNI 192.168.203.124@tcp41 [8/256/0/180] [13859.914621] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13859.969140] LNet: Added LNI 192.168.203.124@tcp42 [8/256/0/180] [13867.937870] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13867.965536] LNet: Added LNI 192.168.203.124@tcp43 [8/256/0/180] [13875.652823] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13875.704224] LNet: Added LNI 192.168.203.124@tcp44 [8/256/0/180] [13884.220931] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13884.264992] LNet: Added LNI 192.168.203.124@tcp45 [8/256/0/180] [13892.343076] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13892.380109] LNet: Added LNI 192.168.203.124@tcp46 [8/256/0/180] [13899.914187] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13899.981462] LNet: Added LNI 192.168.203.124@tcp47 [8/256/0/180] [13907.076588] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13907.102702] LNet: Added LNI 192.168.203.124@tcp48 [8/256/0/180] [13914.973497] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13915.003800] LNet: Added LNI 192.168.203.124@tcp49 [8/256/0/180] [13923.044297] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13923.078243] LNet: Added LNI 192.168.203.124@tcp50 [8/256/0/180] [13939.684720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13939.693757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13939.711222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13941.734452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13941.734831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13941.757431] Lustre: Skipped 2 previous similar messages [13945.790245] Lustre: server umount lustre-MDT0000 complete [13949.278465] LustreError: 473969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786641029 with bad export cookie 16886747822038954699 [13949.293054] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13949.296624] LustreError: 473969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13949.683682] Lustre: server umount lustre-MDT0001 complete [13968.351540] Lustre: 472447:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786641032/real 1786641032] req@000000003495e8c8 x1873428533643904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786641048 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13968.373658] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13968.391762] Lustre: Skipped 1 previous similar message [13969.008984] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [13970.463254] Lustre: 472447:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786641035/real 1786641035] req@00000000f1502cc2 x1873428533644032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786641051 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13972.447110] Lustre: 472445:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786641037/real 1786641037] req@00000000294fe14c x1873428533644160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786641053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13974.559097] Lustre: 472445:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786641039/real 1786641039] req@00000000445fc8b5 x1873428533644416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786641055 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13977.851049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13979.615132] Lustre: 472446:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786641044/real 1786641044] req@000000004d6bcba2 x1873428533644544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786641060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13994.295646] LustreError: 475134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13994.319326] LustreError: 475134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13994.348864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13996.383827] LustreError: 475135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13996.389561] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13998.298400] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [13999.396095] LustreError: 475134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14004.707769] LustreError: 475135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14005.776367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14006.032543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14010.131842] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [14013.430987] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14016.553453] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14021.552799] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14039.012670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14039.013455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14039.035336] Lustre: Skipped 1 previous similar message [14039.060945] Lustre: Skipped 1 previous similar message [14044.132148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14044.145829] Lustre: Skipped 1 previous similar message [14044.580707] Lustre: server umount lustre-OST0000 complete [14049.251551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14049.265855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14049.272577] Lustre: Skipped 1 previous similar message [14054.770606] Lustre: server umount lustre-MDT0000 complete [14057.866557] LustreError: 473167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786641138 with bad export cookie 16886747822038957352 [14057.885483] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14057.889294] LustreError: 473167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14058.270197] Lustre: server umount lustre-MDT0001 complete [14069.242380] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [14071.559308] Key type lgssc unregistered [14071.853373] LNet: 502161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14071.864606] LNetError: 502161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14071.876314] LNet: Removed LNI 192.168.203.124@tcp [14071.879708] LNet: Removed LNI 192.168.203.124@tcp10 [14071.881758] LNet: Removed LNI 192.168.203.124@tcp11 [14071.888650] LNet: Removed LNI 192.168.203.124@tcp12 [14071.891741] LNet: Removed LNI 192.168.203.124@tcp13 [14071.894608] LNet: Removed LNI 192.168.203.124@tcp14 [14071.898706] LNet: Removed LNI 192.168.203.124@tcp15 [14071.903826] LNet: Removed LNI 192.168.203.124@tcp16 [14071.906966] LNet: Removed LNI 192.168.203.124@tcp17 [14071.910147] LNet: Removed LNI 192.168.203.124@tcp18 [14071.918392] LNet: Removed LNI 192.168.203.124@tcp19 [14071.926941] LNet: Removed LNI 192.168.203.124@tcp20 [14071.932924] LNet: Removed LNI 192.168.203.124@tcp21 [14071.936580] LNet: Removed LNI 192.168.203.124@tcp22 [14071.942506] LNet: Removed LNI 192.168.203.124@tcp23 [14071.950154] LNet: Removed LNI 192.168.203.124@tcp24 [14071.953675] LNet: Removed LNI 192.168.203.124@tcp25 [14071.957558] LNet: Removed LNI 192.168.203.124@tcp26 [14071.961854] LNet: Removed LNI 192.168.203.124@tcp27 [14071.966252] LNet: Removed LNI 192.168.203.124@tcp28 [14071.969828] LNet: Removed LNI 192.168.203.124@tcp29 [14071.972891] LNet: Removed LNI 192.168.203.124@tcp30 [14071.976375] LNet: Removed LNI 192.168.203.124@tcp31 [14071.978468] LNet: Removed LNI 192.168.203.124@tcp32 [14071.981227] LNet: Removed LNI 192.168.203.124@tcp33 [14071.982755] LNet: Removed LNI 192.168.203.124@tcp34 [14071.985685] LNet: Removed LNI 192.168.203.124@tcp35 [14071.987687] LNet: Removed LNI 192.168.203.124@tcp36 [14071.990970] LNet: Removed LNI 192.168.203.124@tcp37 [14071.994518] LNet: Removed LNI 192.168.203.124@tcp38 [14071.997863] LNet: Removed LNI 192.168.203.124@tcp39 [14072.000303] LNet: Removed LNI 192.168.203.124@tcp40 [14072.002054] LNet: Removed LNI 192.168.203.124@tcp41 [14072.003247] LNet: Removed LNI 192.168.203.124@tcp42 [14072.004510] LNet: Removed LNI 192.168.203.124@tcp43 [14072.006473] LNet: Removed LNI 192.168.203.124@tcp44 [14072.009141] LNet: Removed LNI 192.168.203.124@tcp45 [14072.010921] LNet: Removed LNI 192.168.203.124@tcp46 [14072.013262] LNet: Removed LNI 192.168.203.124@tcp47 [14072.016425] LNet: Removed LNI 192.168.203.124@tcp48 [14072.018528] LNet: Removed LNI 192.168.203.124@tcp49 [14072.031410] LNet: Removed LNI 192.168.203.124@tcp50 [14072.682535] Key type .llcrypt unregistered [14072.686587] Key type ._llcrypt unregistered [14091.492796] Key type ._llcrypt registered [14091.497614] Key type .llcrypt registered [14091.624392] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [14106.122339] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [14107.288939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14107.462098] alg: No test for adler32 (adler32-zlib) [14108.634693] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [14109.014387] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [14110.687497] Key type lgssc registered [14111.951511] Lustre: Echo OBD driver; http://www.lustre.org/ [14146.033578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14146.044153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14147.273948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14147.323410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14147.444124] Lustre: lustre-MDT0000: new disk, initializing [14147.516159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14147.526987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14151.415402] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [14161.058924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14161.211481] Lustre: 505360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14161.283642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14161.287251] Lustre: Skipped 1 previous similar message [14161.359234] Lustre: lustre-MDT0001: new disk, initializing [14161.447432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14161.476727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14161.493419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14164.535854] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [14171.489500] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14175.759293] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14182.678203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14182.862404] Lustre: *** cfs_fail_loc=90f, val=0*** [14182.916969] Lustre: lustre-OST0000: new disk, initializing [14182.921518] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14182.926550] Lustre: 507374:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14183.033512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14189.377229] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [14191.699984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14191.713792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14191.840940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14198.197597] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14207.461617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14207.464992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14207.477728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14207.494927] Lustre: Skipped 1 previous similar message [14212.583132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14212.592868] Lustre: Skipped 1 previous similar message [14217.702483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14217.710384] Lustre: Skipped 1 previous similar message [14219.231127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14219.357948] Lustre: server umount lustre-OST0000 complete [14222.816412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14222.820312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14222.836753] Lustre: Skipped 2 previous similar messages [14222.842374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14222.850086] Lustre: Skipped 1 previous similar message [14222.871702] LustreError: Skipped 1 previous similar message [14227.940668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14227.953075] Lustre: Skipped 1 previous similar message [14228.896529] Lustre: server umount lustre-MDT0000 complete [14233.484584] LustreError: 507373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786641314 with bad export cookie 2346652687206721667 [14233.489428] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14233.499691] LustreError: 507373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14233.888688] Lustre: server umount lustre-MDT0001 complete [14244.599394] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [14246.902461] Key type lgssc unregistered [14247.163547] LNet: 509284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14247.173692] LNetError: 509284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14247.197803] LNet: Removed LNI 192.168.203.124@tcp [14248.050682] Key type .llcrypt unregistered [14248.052455] Key type ._llcrypt unregistered [14263.317427] Key type ._llcrypt registered [14263.320690] Key type .llcrypt registered [14263.430456] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [14264.080489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14264.103483] alg: No test for adler32 (adler32-zlib) [14265.138825] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [14265.369893] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [14267.032053] Key type lgssc registered [14268.037170] Lustre: Echo OBD driver; http://www.lustre.org/ [14274.681515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14274.708672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14276.216790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14280.538430] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [14285.795381] LustreError: 510533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14285.810873] LustreError: 510533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14288.045613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14288.389269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14292.365382] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [14299.339924] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14304.679869] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14310.589671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14310.814727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14317.179695] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [14325.659658] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14336.368275] loop0: detected capacity change from 0 to 204800000 [14336.528457] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14336.541526] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14336.565106] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14342.282111] loop0: detected capacity change from 0 to 204800000 [14342.564727] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14342.577479] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14342.586782] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14347.101579] loop0: detected capacity change from 0 to 204800000 [14352.062211] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14352.160917] Lustre: fs15246-OST0000: new disk, initializing [14352.164716] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14352.169947] Lustre: 514129:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14352.233892] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14357.768758] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [14366.745936] loop1: detected capacity change from 0 to 204800000 [14371.708591] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14371.816259] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14371.871973] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14371.966308] Lustre: fs15246-MDT0000: new disk, initializing [14372.084544] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14372.107226] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14372.201449] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14373.218792] Lustre: 515252:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000ff229b00 x1873429236068352/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786641469 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14376.569689] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [14382.308880] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14382.322070] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14382.348742] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14395.882660] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14395.897098] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14400.780405] Lustre: server umount fs15246-MDT0000 complete [14400.995523] LustreError: 510529:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14406.130219] LustreError: 510529:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14409.587770] Lustre: server umount fs15246-OST0000 complete [14421.480578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14421.488151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14421.521367] Lustre: Skipped 1 previous similar message [14426.599252] LustreError: 512355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14426.617378] LustreError: 512355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14426.699712] Lustre: server umount lustre-OST0000 complete [14431.713775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14431.717767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14431.720690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14431.723944] Lustre: Skipped 1 previous similar message [14431.738536] Lustre: Skipped 2 previous similar messages [14436.285278] Lustre: server umount lustre-MDT0000 complete [14436.832675] LustreError: 510534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14436.855036] LustreError: 510534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14439.346173] LustreError: 510514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786641519 with bad export cookie 11123362739881892544 [14439.350777] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14439.356365] LustreError: 510514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14439.698890] Lustre: server umount lustre-MDT0001 complete [14450.969692] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [14454.247679] Key type lgssc unregistered [14454.547072] LNet: 518351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14454.553013] LNetError: 518351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14454.577439] LNet: Removed LNI 192.168.203.124@tcp [14455.429570] Key type .llcrypt unregistered [14455.431599] Key type ._llcrypt unregistered [14474.704414] Key type ._llcrypt registered [14474.706925] Key type .llcrypt registered [14474.809544] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [14475.979274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14476.013116] alg: No test for adler32 (adler32-zlib) [14477.186870] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [14477.483301] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [14479.183157] Key type lgssc registered [14480.387619] Lustre: Echo OBD driver; http://www.lustre.org/ [14519.105567] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [14530.986763] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [14571.847581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14571.869259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14573.093561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14573.138473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14573.237874] Lustre: lustre-MDT0000: new disk, initializing [14573.355550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14573.384122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14578.458441] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [14588.747373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14588.877952] Lustre: 522820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14588.966057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14588.970974] Lustre: Skipped 1 previous similar message [14589.074302] Lustre: lustre-MDT0001: new disk, initializing [14589.181537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14589.217248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14589.227170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14593.394837] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [14602.971912] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14608.465139] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14615.109971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14615.459586] Lustre: lustre-OST0000: new disk, initializing [14615.462604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14615.469440] Lustre: 524788:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14615.549764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14621.705300] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [14622.812933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14622.825871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14622.952920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14630.281409] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14634.599754] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14634.808371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14639.321200] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14639.567331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14643.685462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14643.689614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14643.701357] Lustre: Skipped 1 previous similar message [14647.624484] Lustre: server umount lustre-OST0000 complete [14653.929682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14653.938231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14653.951302] Lustre: Skipped 1 previous similar message [14653.968602] Lustre: Skipped 2 previous similar messages [14657.755296] Lustre: server umount lustre-MDT0000 complete [14661.638273] LustreError: 522810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786641742 with bad export cookie 1736913671283105246 [14661.641285] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14661.649589] LustreError: 522810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14662.106501] Lustre: server umount lustre-MDT0001 complete [14685.862700] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [14695.621959] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [14727.782697] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14490 sec ========= 13:23:26 (1786641806) [14729.600273] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:23:28 (1786641808) === [14732.831483] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:23:31 (1786641811) === [14769.509952] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [14772.355070] Key type lgssc unregistered [14772.730044] LNet: 531432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14772.742214] LNetError: 531432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14773.797663] LNet: Removed LNI 192.168.203.124@tcp [14774.911469] Key type .llcrypt unregistered [14774.915025] Key type ._llcrypt unregistered