[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 509084219 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002423] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008019] pid_max: default: 32768 minimum: 301 [ 0.009136] LSM: Security Framework initializing [ 0.010064] Yama: becoming mindful. [ 0.011054] SELinux: Initializing. [ 0.013065] *** VALIDATE selinux *** [ 0.022475] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027873] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030138] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031127] *** VALIDATE tmpfs *** [ 0.033039] *** VALIDATE proc *** [ 0.034323] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036313] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039038] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043905] debug: unmapping init [mem 0xffffffffbb459000-0xffffffffbb460fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046755] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049017] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051019] ... max period: 00007fffffffffff [ 0.052021] ... fixed-purpose events: 3 [ 0.053017] ... event mask: 000000070000000f [ 0.054310] rcu: Hierarchical SRCU implementation. [ 0.056559] smp: Bringing up secondary CPUs ... [ 0.057670] x86: Booting SMP configuration: [ 0.058032] .... node #0, CPUs: #1 #2 #3 [ 0.062577] smp: Brought up 1 node, 4 CPUs [ 0.064020] smpboot: Max logical packages: 1 [ 0.065016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.241035] node 0 deferred pages initialised in 174ms [ 0.245017] devtmpfs: initialized [ 0.246050] x86/mm: Memory block size: 128MB [ 0.248624] gcov: version magic: 0x41383552 [ 0.251284] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.252085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.253263] pinctrl core: initialized pinctrl subsystem [ 0.254130] [ 0.254759] ************************************************************* [ 0.257024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258000] ** ** [ 0.260013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.262016] ** ** [ 0.264015] ** This means that this kernel is built to expose internal ** [ 0.266016] ** IOMMU data structures, which may compromise security on ** [ 0.268015] ** your system. ** [ 0.270014] ** ** [ 0.273019] ** If you see this message and you are not debugging the ** [ 0.275016] ** kernel, report this immediately to your vendor! ** [ 0.278017] ** ** [ 0.280015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.282015] ************************************************************* [ 0.285723] NET: Registered protocol family 16 [ 0.287439] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.290129] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.292088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.296057] cpuidle: using governor menu [ 0.298495] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.300626] PCI: Using configuration type 1 for base access [ 0.303171] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.311126] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.313045] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.316092] cryptd: max_cpu_qlen set to 1000 [ 0.318284] ACPI: Added _OSI(Module Device) [ 0.320020] ACPI: Added _OSI(Processor Device) [ 0.321013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.322015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.326000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.335660] ACPI: Interpreter enabled [ 0.337136] ACPI: PM: (supports S0 S3 S4 S5) [ 0.338027] ACPI: Using IOAPIC for interrupt routing [ 0.340174] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.344447] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.356312] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.358042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.361023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.363080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.368701] acpiphp: Slot [2] registered [ 0.370152] acpiphp: Slot [5] registered [ 0.371186] acpiphp: Slot [6] registered [ 0.372172] acpiphp: Slot [7] registered [ 0.373153] acpiphp: Slot [8] registered [ 0.375127] acpiphp: Slot [9] registered [ 0.376166] acpiphp: Slot [10] registered [ 0.377182] acpiphp: Slot [3] registered [ 0.379110] acpiphp: Slot [4] registered [ 0.380151] acpiphp: Slot [11] registered [ 0.381122] acpiphp: Slot [12] registered [ 0.382099] acpiphp: Slot [13] registered [ 0.383136] acpiphp: Slot [14] registered [ 0.385216] acpiphp: Slot [15] registered [ 0.386128] acpiphp: Slot [16] registered [ 0.388147] acpiphp: Slot [17] registered [ 0.390215] acpiphp: Slot [18] registered [ 0.391137] acpiphp: Slot [19] registered [ 0.392203] acpiphp: Slot [20] registered [ 0.394211] acpiphp: Slot [21] registered [ 0.396161] acpiphp: Slot [22] registered [ 0.398136] acpiphp: Slot [23] registered [ 0.400139] acpiphp: Slot [24] registered [ 0.402144] acpiphp: Slot [25] registered [ 0.403195] acpiphp: Slot [26] registered [ 0.405141] acpiphp: Slot [27] registered [ 0.407118] acpiphp: Slot [28] registered [ 0.409135] acpiphp: Slot [29] registered [ 0.411120] acpiphp: Slot [30] registered [ 0.412121] acpiphp: Slot [31] registered [ 0.414077] PCI host bridge to bus 0000:00 [ 0.416026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.419027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.421028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.424075] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.427026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.430028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.432162] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.435169] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.439678] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.448939] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.454149] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.456014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.458017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.460016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.463099] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.464931] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.468050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.470943] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.476016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.488018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.494016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.499000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.507016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.514019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.530019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.539964] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.551018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.557020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.575020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.593085] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.602018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.611018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.628021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.637626] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.650019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.657017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.686020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.694825] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.713020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.720020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.737000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.749339] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.756019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.765018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.787021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.800605] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.803420] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.805450] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.808466] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.810158] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.814118] iommu: Default domain type: Passthrough [ 0.816489] SCSI subsystem initialized [ 0.818170] ACPI: bus type USB registered [ 0.819140] usbcore: registered new interface driver usbfs [ 0.822096] usbcore: registered new interface driver hub [ 0.824138] usbcore: registered new device driver usb [ 0.826208] pps_core: LinuxPPS API ver. 1 registered [ 0.828012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.831093] PTP clock support registered [ 0.834057] EDAC MC: Ver: 3.0.0 [ 0.836188] PCI: Using ACPI for IRQ routing [ 0.837550] NetLabel: Initializing [ 0.839014] NetLabel: domain hash size = 128 [ 0.841011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.843097] NetLabel: unlabeled traffic allowed by default [ 0.845334] vgaarb: loaded [ 0.846277] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.848015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.856834] clocksource: Switched to clocksource kvm-clock [ 0.966596] VFS: Disk quotas dquot_6.6.0 [ 0.969801] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.971862] *** VALIDATE ramfs *** [ 0.973444] *** VALIDATE hugetlbfs *** [ 0.975166] pnp: PnP ACPI init [ 0.977564] pnp: PnP ACPI: found 6 devices [ 0.998321] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.003990] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.007606] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.009734] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.011877] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.014115] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.017088] NET: Registered protocol family 2 [ 1.019742] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.024734] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.027984] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.033061] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.036849] TCP: Hash tables configured (established 65536 bind 65536) [ 1.040123] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.044056] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.046979] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.050115] NET: Registered protocol family 1 [ 1.053637] RPC: Registered named UNIX socket transport module. [ 1.055755] RPC: Registered udp transport module. [ 1.057421] RPC: Registered tcp transport module. [ 1.058971] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.060786] NET: Registered protocol family 44 [ 1.061941] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.063403] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.064755] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.066672] PCI: CLS 0 bytes, default 64 [ 1.067819] Unpacking initramfs... [ 2.555662] debug: unmapping init [mem 0xffff9863bcc54000-0xffff9863bffbffff] [ 2.562165] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.564822] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.568792] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.110364] Initialise system trusted keyrings [ 3.113029] Key type blacklist registered [ 3.115302] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.126176] zbud: loaded [ 3.129542] *** VALIDATE nfs *** [ 3.130911] *** VALIDATE nfs4 *** [ 3.132644] pstore: using deflate compression [ 3.136936] Platform Keyring initialized [ 3.267756] NET: Registered protocol family 38 [ 3.269704] Key type asymmetric registered [ 3.271306] Asymmetric key parser 'x509' registered [ 3.273423] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.277133] io scheduler mq-deadline registered [ 3.279397] io scheduler kyber registered [ 3.281228] io scheduler bfq registered [ 3.283408] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.286838] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.289827] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.292967] ACPI: Power Button [PWRF] [ 3.299335] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.306248] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.325860] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.335258] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.351984] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.382652] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.413797] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.419168] Non-volatile memory driver v1.3 [ 3.421036] Linux agpgart interface v0.103 [ 3.454879] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.457482] vda: detected capacity change from 0 to 76677120 [ 3.482063] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.485482] vdb: detected capacity change from 0 to 1073741824 [ 3.501831] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.504899] vdc: detected capacity change from 0 to 2621440000 [ 3.524148] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.527687] vdd: detected capacity change from 0 to 2621440000 [ 3.544451] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.547530] vde: detected capacity change from 0 to 4294967296 [ 3.566784] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.570676] vdf: detected capacity change from 0 to 4294967296 [ 3.579322] libphy: Fixed MDIO Bus: probed [ 3.585210] usbcore: registered new interface driver usbserial_generic [ 3.587756] usbserial: USB Serial support registered for generic [ 3.590336] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.595508] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.597760] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.601681] mousedev: PS/2 mouse device common for all mice [ 3.606213] rtc_cmos 00:05: RTC can wake from S4 [ 3.613489] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.613576] rtc_cmos 00:05: registered as rtc0 [ 3.619366] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.620552] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.622478] intel_pstate: CPU model not supported [ 3.627793] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.632673] hid: raw HID events driver (C) Jiri Kosina [ 3.634663] usbcore: registered new interface driver usbhid [ 3.636681] usbhid: USB HID core driver [ 3.638270] drop_monitor: Initializing network drop monitor service [ 3.640474] Initializing XFRM netlink socket [ 3.642944] NET: Registered protocol family 10 [ 3.646101] Segment Routing with IPv6 [ 3.647522] NET: Registered protocol family 17 [ 3.649522] mpls_gso: MPLS GSO support [ 3.656636] RAS: Correctable Errors collector initialized. [ 3.658702] AVX version of gcm_enc/dec engaged. [ 3.660269] AES CTR mode by8 optimization enabled [ 3.739956] sched_clock: Marking stable (3739908338, 0)->(4686665827, -946757489) [ 3.743493] registered taskstats version 1 [ 3.745673] Loading compiled-in X.509 certificates [ 3.748051] zswap: loaded using pool lzo/zbud [ 3.775979] Key type big_key registered [ 3.788669] Key type encrypted registered [ 3.790498] ima: No TPM chip found, activating TPM-bypass! [ 3.792319] ima: Allocated hash algorithm: sha1 [ 3.794165] ima: No architecture policies found [ 3.796139] evm: Initialising EVM extended attributes: [ 3.798053] evm: security.selinux [ 3.799406] evm: security.ima [ 3.800614] evm: security.capability [ 3.802177] evm: HMAC attrs: 0x1 [ 3.804759] rtc_cmos 00:05: setting system clock to 2026-09-03 12:20:31 UTC (1788438031) [ 3.813519] debug: unmapping init [mem 0xffffffffbc403000-0xffffffffbc5fffff] [ 3.816685] debug: unmapping init [mem 0xffffffffbb182000-0xffffffffbb458fff] [ 3.828083] Write protecting the kernel read-only data: 28672k [ 3.831994] debug: unmapping init [mem 0xffffffffb9803000-0xffffffffb99fffff] [ 3.834919] debug: unmapping init [mem 0xffffffffba114000-0xffffffffba1fffff] [ 3.872996] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.883370] systemd[1]: Detected virtualization kvm. [ 3.885263] systemd[1]: Detected architecture x86-64. [ 3.887276] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.910360] systemd[1]: No hostname configured. [ 3.912426] systemd[1]: Set hostname to . [ 3.914154] random: systemd: uninitialized urandom read (16 bytes read) [ 3.917281] systemd[1]: Initializing machine ID from random generator. [ 4.042388] random: systemd: uninitialized urandom read (16 bytes read) [ 4.044794] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.050604] random: systemd: uninitialized urandom read (16 bytes read) [ 4.053412] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.058160] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.752332] device-mapper: uevent: version 1.0.3 [ 4.754787] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.456470] virtio_net virtio0 ens2: renamed from eth0 [ 5.456625] random: fast init done [ 5.525608] scsi host0: ata_piix [ 5.570954] scsi host1: ata_piix [ 5.573226] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.575814] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.837600] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.309356] random: crng init done [ 10.310867] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.763162] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.976952] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.264053] SELinux: Disabled at runtime. [ 12.331554] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.341043] systemd[1]: Detected virtualization kvm. [ 12.342680] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.044298] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.053935] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.066527] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.073304] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.081393] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.095034] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.103041] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ 13.485285] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 14.912179] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.007711] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.091677] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.685755] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.734161] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 23.316302] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (10s / no limit)[ 24.297208] NFS: Registering the id_resolver key type [ 24.300175] Key type id_resolver registered [ 24.303445] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Hostname Service... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg421-server login: [ 80.402919] libcfs: loading out-of-tree module taints kernel. [ 80.455568] Key type ._llcrypt registered [ 80.458077] Key type .llcrypt registered [ 80.576193] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 97.235768] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 98.496404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.521913] alg: No test for adler32 (adler32-zlib) [ 99.863897] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 100.632330] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 102.352758] Key type lgssc registered [ 103.945822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 120.445667] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 163.821448] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 174.985041] hrtimer: interrupt took 1982701 ns [ 179.928963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 179.968135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.210629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 181.240944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 181.317943] Lustre: lustre-MDT0000: new disk, initializing [ 181.404574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.427779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 185.732118] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.248881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.402799] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 199.460960] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 199.468381] Lustre: Skipped 1 previous similar message [ 199.539901] Lustre: lustre-MDT0001: new disk, initializing [ 199.619428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.653042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 199.676699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 204.346799] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.564400] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 219.599482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.867960] Lustre: lustre-OST0000: new disk, initializing [ 219.881435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 219.898539] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.979471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 226.080864] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.909523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 227.927018] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 228.023886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 239.522877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.646702] Lustre: lustre-OST0001: new disk, initializing [ 239.655173] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 239.664704] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.722487] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 245.754718] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.407376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 249.417720] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 249.479306] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 256.564920] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 261.621409] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 267.168845] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing check_logdir /tmp/testlogs/ [ 272.585279] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing yml_node [ 277.650890] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 280.276941] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 283.042361] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 284.725907] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:25:11 EDT 2026 [ 301.597973] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 303.264341] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 320.993747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.001790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 321.016871] Lustre: Skipped 2 previous similar messages [ 321.037287] Lustre: Skipped 3 previous similar messages [ 325.516340] Lustre: server umount lustre-MDT0000 complete [ 329.206799] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438356 with bad export cookie 14514076399618404871 [ 329.210786] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 329.224364] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 331.233448] LustreError: 11375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.233934] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.246042] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 331.249174] LustreError: 11375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 331.278516] Lustre: Skipped 2 previous similar messages [ 334.241154] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 334.241204] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 334.262250] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 334.277313] Lustre: Skipped 2 previous similar messages [ 335.682188] Lustre: server umount lustre-MDT0001 complete [ 339.904418] Lustre: server umount lustre-OST0000 complete [ 343.884513] Lustre: server umount lustre-OST0001 complete [ 351.208760] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 362.304706] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 400.671520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.951494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 400.999640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 401.073730] Lustre: lustre-MDT0000: new disk, initializing [ 401.188950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.207563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 406.151846] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 417.425123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.551322] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 417.634388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 417.641659] Lustre: Skipped 1 previous similar message [ 417.725528] Lustre: lustre-MDT0001: new disk, initializing [ 417.843671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 417.897266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 417.917237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 422.553906] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 433.312488] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 439.598374] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 447.336121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 447.783371] Lustre: lustre-OST0000: new disk, initializing [ 447.794950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 447.804850] Lustre: 17048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 447.904596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.857744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 448.862515] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 449.000437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 454.358896] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 464.661490] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 469.922607] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 470.223455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 475.485851] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 475.852747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 479.716932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.731438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 479.772850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 479.781371] Lustre: Skipped 1 previous similar message [ 484.351681] Lustre: server umount lustre-OST0000 complete [ 489.443031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 489.461807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.478349] Lustre: Skipped 1 previous similar message [ 489.487973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 495.031160] Lustre: server umount lustre-MDT0000 complete [ 495.080992] LustreError: 15944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.111644] LustreError: 15944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 499.649465] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438527 with bad export cookie 14514076399618406404 [ 499.650477] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.654485] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 499.913571] Lustre: server umount lustre-MDT0001 complete [ 505.617473] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:28:51 (1788438531) [ 510.021356] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 512.196754] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:28:58 (1788438538) [ 520.321799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.995645] LustreError: 19458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.101184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 525.947205] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 526.308165] LustreError: 19459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.319563] LustreError: 19459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 534.414048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 539.808344] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 547.258927] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 552.145144] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 558.461584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 564.121584] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 570.270781] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 576.486245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 576.497520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 576.501847] Lustre: Skipped 1 previous similar message [ 576.513156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 576.518607] Lustre: Skipped 2 previous similar messages [ 582.621489] Lustre: server umount lustre-OST0000 complete [ 586.214241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 586.220193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.411943] Lustre: server umount lustre-MDT0000 complete [ 595.527820] LustreError: 19439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438623 with bad export cookie 14514076399618407286 [ 595.532690] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.538992] LustreError: 19439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 595.848887] Lustre: server umount lustre-MDT0001 complete [ 605.921293] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 608.683668] Key type lgssc unregistered [ 608.921798] LNet: 22987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.927719] LNetError: 22987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 608.945989] LNet: Removed LNI 192.168.204.121@tcp [ 609.748177] Key type .llcrypt unregistered [ 609.752540] Key type ._llcrypt unregistered [ 631.001630] Key type ._llcrypt registered [ 631.004511] Key type .llcrypt registered [ 631.111944] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 632.619925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 632.642412] alg: No test for adler32 (adler32-zlib) [ 633.798297] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 634.058293] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 635.760166] Key type lgssc registered [ 637.179976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 644.733379] loop: module loaded [ 644.783957] loop0: detected capacity change from 0 to 204800000 [ 644.964815] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.977168] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 645.289812] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.367216] loop0: detected capacity change from 0 to 204800000 [ 651.613979] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.629665] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.660991] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.870337] loop0: detected capacity change from 0 to 204800000 [ 662.107633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 662.126658] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.387805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 663.438920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 663.501647] Lustre: lustre-MDT0000: new disk, initializing [ 663.608461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 663.629274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 667.930148] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 676.046841] loop1: detected capacity change from 0 to 204800000 [ 680.820911] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 681.073231] Lustre: lustre-OST07c6: new disk, initializing [ 681.077210] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 681.081652] Lustre: Skipped 1 previous similar message [ 681.090306] Lustre: 26174:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 681.151525] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 681.601273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 681.615532] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 681.656382] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 685.827279] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 690.741469] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 690.759227] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 690.769334] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 690.777089] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 690.786920] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 690.792437] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 712.119047] Lustre: 26742:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 712.143849] Lustre: 26742:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 712.151190] Lustre: 26742:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 712.169716] Lustre: 26742:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 712.182608] Lustre: 26742:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 712.194217] Lustre: 26742:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 716.975159] Lustre: 26742:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 717.002397] Lustre: 26742:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 717.024427] Lustre: 26742:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 717.037559] Lustre: 26742:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.052751] Lustre: 26742:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 717.061824] Lustre: 26742:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.069526] Lustre: 26742:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 717.083788] Lustre: 26742:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.101520] Lustre: 26742:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 717.108196] Lustre: 26742:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.115852] Lustre: 26742:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 717.136577] Lustre: 26742:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.354373] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 720.360195] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 720.366119] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 720.371382] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.376982] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 720.381795] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.385540] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 720.392940] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.401362] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 720.409575] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.415664] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 720.420884] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.087687] Lustre: 25161:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 725.102822] Lustre: 25161:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 725.111557] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 725.119867] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.129128] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 725.139441] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.146324] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 725.157415] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.167267] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 725.178749] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.185912] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.190623] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 736.177759] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 736.193925] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 736.205369] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 736.216033] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 736.222653] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 736.232491] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 736.239755] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 736.249675] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 736.256629] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 736.262524] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 736.269294] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 736.274904] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 788.972137] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 788.984151] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 788.987458] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 794.085090] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 798.176196] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 798.330804] Lustre: server umount lustre-OST07c6 complete [ 806.994107] Lustre: server umount lustre-MDT0000 complete [ 824.147804] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 827.194632] Key type lgssc unregistered [ 827.537854] LNet: 28732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.547387] LNetError: 28732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.563275] LNet: Removed LNI 192.168.204.121@tcp [ 828.541644] Key type .llcrypt unregistered [ 828.549507] Key type ._llcrypt unregistered [ 847.413910] Key type ._llcrypt registered [ 847.416264] Key type .llcrypt registered [ 847.560754] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 848.600345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 848.807773] alg: No test for adler32 (adler32-zlib) [ 849.928514] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 850.131856] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 851.840232] Key type lgssc registered [ 852.885749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 861.446897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 861.500336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.037822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 868.010755] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 872.424662] LustreError: 30000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 872.433208] LustreError: 30000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 877.513343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.537983] LustreError: 30001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 877.995671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 883.866993] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 891.716185] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 897.862572] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 904.970221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 905.523171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.586453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 911.817749] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 919.129501] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 940.953557] Lustre: server umount lustre-OST0000 complete [ 948.708897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.714305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 948.722292] Lustre: Skipped 1 previous similar message [ 953.828395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 953.837334] Lustre: Skipped 2 previous similar messages [ 954.982545] Lustre: server umount lustre-MDT0000 complete [ 958.968638] LustreError: 30940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 959.000345] LustreError: 30940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 959.834947] LustreError: 29982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438987 with bad export cookie 7705616903836893088 [ 959.866264] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 970.390054] Lustre: server umount lustre-MDT0001 complete [ 981.877485] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 984.948476] Key type lgssc unregistered [ 985.262481] LNet: 33739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.272967] LNetError: 33739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.300877] LNet: Removed LNI 192.168.204.121@tcp [ 986.178126] Key type .llcrypt unregistered [ 986.180773] Key type ._llcrypt unregistered [ 1003.829965] Key type ._llcrypt registered [ 1003.832899] Key type .llcrypt registered [ 1003.963790] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1004.721622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1004.765378] alg: No test for adler32 (adler32-zlib) [ 1005.779752] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1005.970679] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1007.600172] Key type lgssc registered [ 1008.655894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1016.109914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1016.135931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1017.652978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1022.495689] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1028.068635] LustreError: 35021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1028.097587] LustreError: 35021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1031.253451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.599193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1035.728888] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1042.658848] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1048.262223] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1055.335931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1055.664597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1062.122548] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1062.853896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1069.635771] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1082.340187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1082.347562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.356321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1083.361948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1083.371709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1083.405708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1085.985595] Lustre: server umount lustre-OST0000 complete [ 1093.092454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1093.099837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.117316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1093.130139] Lustre: Skipped 1 previous similar message [ 1098.727637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1098.734992] Lustre: Skipped 2 previous similar messages [ 1103.843775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1103.849903] Lustre: Skipped 1 previous similar message [ 1104.352464] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1104.620222] Lustre: server umount lustre-MDT0000 complete [ 1108.441327] LustreError: 35001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439136 with bad export cookie 8531817392453789191 [ 1108.450725] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1108.478556] LustreError: 35001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1108.836214] Lustre: server umount lustre-MDT0001 complete [ 1121.333197] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1124.036710] Key type lgssc unregistered [ 1124.299280] LNet: 38557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1124.304827] LNetError: 38557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1124.322202] LNet: Removed LNI 192.168.204.121@tcp [ 1125.130237] Key type .llcrypt unregistered [ 1125.133305] Key type ._llcrypt unregistered [ 1147.521288] Key type ._llcrypt registered [ 1147.523238] Key type .llcrypt registered [ 1147.643542] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1148.967659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1149.083144] alg: No test for adler32 (adler32-zlib) [ 1150.110993] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1150.477458] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1152.176196] Key type lgssc registered [ 1153.402494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1162.319407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1162.355773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1163.993131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1169.548902] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1173.474972] LustreError: 39951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1173.487356] LustreError: 39951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1178.594331] LustreError: 39952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1180.294926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.790309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1186.188256] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1194.592636] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1201.003393] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1208.768786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1209.227135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1216.851495] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1226.423866] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1233.775964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1243.390318] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.21@tcp (stopping) [ 1243.620877] Lustre: server umount lustre-MDT0000 complete [ 1244.136057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1244.150704] Lustre: Skipped 1 previous similar message [ 1244.156654] LustreError: 39951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.167373] LustreError: 39951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1247.141053] LustreError: 39931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439274 with bad export cookie 12859081061891165944 [ 1247.155447] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1247.157683] LustreError: 39931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1248.487113] LustreError: 39946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.512906] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.21@tcp (stopping) [ 1249.251362] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1249.546417] Lustre: server umount lustre-MDT0001 complete [ 1261.807944] LustreError: 39932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.21@tcp arrived at 1788439289 with bad export cookie 12859081061891166609 [ 1261.827706] LustreError: 39932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1270.708226] Lustre: server umount lustre-OST0000 complete [ 1286.437148] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1288.794342] Key type lgssc unregistered [ 1289.048809] LNet: 43831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1289.057263] LNetError: 43831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1289.075267] LNet: Removed LNI 192.168.204.121@tcp [ 1289.796232] Key type .llcrypt unregistered [ 1289.797879] Key type ._llcrypt unregistered [ 1320.403901] Key type ._llcrypt registered [ 1320.405494] Key type .llcrypt registered [ 1320.514280] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1321.577198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1321.675460] alg: No test for adler32 (adler32-zlib) [ 1322.706914] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1322.986468] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1324.720534] Key type lgssc registered [ 1325.849834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1332.982652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1332.994527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.660075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1339.870134] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1345.002737] LustreError: 45243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.020740] LustreError: 45243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1348.413644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.838935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1354.703725] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1363.726326] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1370.036121] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1378.561610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1379.102899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1380.165599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1387.268453] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1396.344784] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1404.157095] Lustre: lustre-OST0000: Not available for connect from 192.168.204.21@tcp (stopping) [ 1404.900614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1404.907883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1404.918406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1404.927342] Lustre: Skipped 1 previous similar message [ 1409.256338] Lustre: lustre-OST0000: Not available for connect from 192.168.204.21@tcp (stopping) [ 1409.267666] Lustre: Skipped 1 previous similar message [ 1410.178276] Lustre: server umount lustre-OST0000 complete [ 1434.594332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.601558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1434.621576] Lustre: Skipped 1 previous similar message [ 1434.627045] Lustre: Skipped 3 previous similar messages [ 1438.055339] Lustre: server umount lustre-MDT0000 complete [ 1439.713042] LustreError: 45243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1439.731193] LustreError: 45243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1442.152143] LustreError: 45225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439469 with bad export cookie 14625149873420307030 [ 1442.164875] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1448.540428] Lustre: server umount lustre-MDT0001 complete [ 1466.275821] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1469.488604] Key type lgssc unregistered [ 1469.967333] LNet: 48957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.974967] LNetError: 48957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.997725] LNet: Removed LNI 192.168.204.121@tcp [ 1470.972091] Key type .llcrypt unregistered [ 1470.975221] Key type ._llcrypt unregistered [ 1501.681802] Key type ._llcrypt registered [ 1501.688588] Key type .llcrypt registered [ 1501.835181] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1503.331730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1503.373093] alg: No test for adler32 (adler32-zlib) [ 1504.620086] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1505.071458] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1506.769047] Key type lgssc registered [ 1508.118741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1517.361319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1517.376328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1518.918646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.474311] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1529.322354] LustreError: 50369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1529.352304] LustreError: 50369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1534.448516] LustreError: 50368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1535.741615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1536.479558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1541.994246] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1550.996085] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1559.872369] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1569.092186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1569.545055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1576.797666] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1585.254743] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1590.004311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1596.039846] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1600.146496] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1613.105983] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 8h47m18s [ 1615.121442] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1617.689374] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.21@tcp (stopping) [ 1618.401441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1618.406323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.416529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1619.508449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1619.511492] Lustre: Skipped 5 previous similar messages [ 1623.507829] Lustre: server umount lustre-MDT0000 complete [ 1623.780253] LustreError: 50365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1628.110861] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1629.926278] LustreError: 50365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1629.954358] LustreError: 50365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1636.070113] LustreError: 50363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1636.081684] LustreError: 50363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1636.433881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1636.629654] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1636.922388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.011192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1637.937460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1642.449553] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1646.768346] Lustre: DEBUG MARKER: Wait for df (36658) ... [ 1648.971271] Lustre: DEBUG MARKER: done [ 1657.315991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1657.329662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.344615] Lustre: Skipped 2 previous similar messages [ 1657.364320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1663.462035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1663.476444] Lustre: Skipped 4 previous similar messages [ 1668.064573] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1668.335469] Lustre: server umount lustre-OST0000 complete [ 1673.702209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.712795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1673.717059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1673.717067] LustreError: Skipped 1 previous similar message [ 1673.721386] Lustre: Skipped 2 previous similar messages [ 1673.772894] Lustre: Skipped 1 previous similar message [ 1679.399949] Lustre: server umount lustre-MDT0000 complete [ 1680.874862] LustreError: 51097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1680.895805] LustreError: 51097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1684.166142] LustreError: 50349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439711 with bad export cookie 13795559880400481518 [ 1684.182974] LustreError: 50349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1684.183770] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1684.701407] Lustre: server umount lustre-MDT0001 complete [ 1698.753894] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1702.138255] Key type lgssc unregistered [ 1702.536446] LNet: 55297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.543842] LNetError: 55297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.559711] LNet: Removed LNI 192.168.204.121@tcp [ 1703.654432] Key type .llcrypt unregistered [ 1703.660777] Key type ._llcrypt unregistered [ 1752.114827] Key type ._llcrypt registered [ 1752.118551] Key type .llcrypt registered [ 1752.285198] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1753.938278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1753.967533] alg: No test for adler32 (adler32-zlib) [ 1755.132576] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1755.344457] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1756.992319] Key type lgssc registered [ 1758.592698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1768.550276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1768.585961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1770.076141] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1770.116345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1770.421333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1775.950064] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1785.269538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.358983] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1785.394663] Lustre: 57507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1785.693144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1790.310138] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1800.704800] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1808.514664] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1817.630526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1818.031438] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1818.380617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.970738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1825.931247] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1834.849659] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1843.187129] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 8h51m09s [ 1845.540623] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1854.041756] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1857.686348] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1857.730663] LustreError: 57512:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1878.320989] Lustre: lustre-MDT0000: Client 7f4db75b-821d-4662-a53b-70dd83643b1f (at 192.168.204.21@tcp) reconnecting [ 1897.776137] LustreError: 57512:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1901.951526] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1905.183139] Lustre: DEBUG MARKER: done [ 1916.394787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1916.395843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1916.428706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1916.437828] Lustre: Skipped 1 previous similar message [ 1918.108527] Lustre: server umount lustre-OST0000 complete [ 1924.065432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1924.076480] LustreError: Skipped 1 previous similar message [ 1924.084324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1924.102184] Lustre: Skipped 1 previous similar message [ 1924.122973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1925.156055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1925.166463] Lustre: Skipped 2 previous similar messages [ 1927.208453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.216837] Lustre: Skipped 1 previous similar message [ 1933.282938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1933.291341] Lustre: Skipped 1 previous similar message [ 1937.376343] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1937.781854] Lustre: server umount lustre-MDT0000 complete [ 1942.810677] LustreError: 59370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439970 with bad export cookie 13372837148265865767 [ 1942.823591] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1942.834528] LustreError: 59370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1943.325487] Lustre: server umount lustre-MDT0001 complete [ 1957.520544] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1960.355591] Key type lgssc unregistered [ 1960.630811] LNet: 61839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.641895] LNetError: 61839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.668953] LNet: Removed LNI 192.168.204.121@tcp [ 1961.484793] Key type .llcrypt unregistered [ 1961.487609] Key type ._llcrypt unregistered [ 2011.401407] Key type ._llcrypt registered [ 2011.404826] Key type .llcrypt registered [ 2011.559681] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2012.911357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2012.931587] alg: No test for adler32 (adler32-zlib) [ 2014.005279] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2014.277469] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2015.993540] Key type lgssc registered [ 2017.546633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2021.406534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2021.436524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2022.707733] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2022.743640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2022.942447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2024.003186] Lustre: Failing over lustre-MDT0000 [ 2024.380537] Lustre: server umount lustre-MDT0000 complete [ 2031.793465] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:54:18 (1788440058) [ 2039.331769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.881061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.536495] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2055.751682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2055.870812] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2055.925470] Lustre: 64637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2056.297995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2061.978706] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2071.728415] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2079.378788] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2089.095605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2089.232435] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2089.551337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.095271] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2099.425513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2108.281948] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2115.250853] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2122.721901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2122.726985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2122.734070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2125.286068] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2125.289984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2125.318907] Lustre: Skipped 2 previous similar messages [ 2128.897162] Lustre: server umount lustre-MDT0000 complete [ 2133.606735] LustreError: 64628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440161 with bad export cookie 311369828511004909 [ 2133.610623] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2133.614680] LustreError: 64628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2134.128375] Lustre: server umount lustre-MDT0001 complete [ 2138.432957] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2143.824991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2151.906500] Lustre: 63564:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440163/real 1788440163] req@000000009cb98aef x1875315304476032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440179 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2151.935467] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.944212] Lustre: Skipped 1 previous similar message [ 2154.626031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2154.976293] Lustre: 63561:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440166/real 1788440166] req@00000000a198ab24 x1875315304476160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440182 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2157.024479] Lustre: 63562:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440168/real 1788440168] req@000000006323322c x1875315304476288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2159.140896] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4523535962c41d0 [ 2159.155829] Lustre: MGC192.168.204.121@tcp: Connection restored to 0@lo (at 0@lo) [ 2159.475953] LustreError: 66490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2159.501374] LustreError: 66490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2159.619111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2159.758227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2160.620806] Lustre: 63563:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440171/real 1788440171] req@00000000965858f2 x1875315304476544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440187 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2161.634195] LustreError: 66491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2161.652903] LustreError: 66491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2164.713600] Lustre: 63564:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440176/real 1788440176] req@000000005898123a x1875315304476672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2165.254172] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2166.753410] LustreError: 68414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.776308] LustreError: 68414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2171.876758] LustreError: 68413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2171.897751] LustreError: 68413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2175.546311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2176.057843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2181.107268] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2182.007792] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2182.201056] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2190.278454] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2196.897914] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2206.697581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2206.713597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.733612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2210.832178] Lustre: server umount lustre-MDT0000 complete [ 2212.847621] LustreError: 68415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.887134] LustreError: 68415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2215.745444] LustreError: 64629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440243 with bad export cookie 311369828511007184 [ 2215.767883] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2215.769192] LustreError: 64629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2216.444368] Lustre: server umount lustre-MDT0001 complete [ 2224.322415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2233.313706] Lustre: 63561:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440245/real 1788440245] req@00000000376cd3ec x1875315304506624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440261 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2233.355127] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.402564] Lustre: Skipped 2 previous similar messages [ 2235.187232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.029410] LustreError: 66491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2241.175985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2241.284715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2246.873973] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2256.982589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2262.506257] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2262.983347] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2271.731265] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2277.879869] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2288.099384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2288.111605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2288.130431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2288.140582] Lustre: Skipped 3 previous similar messages [ 2292.627489] Lustre: server umount lustre-MDT0000 complete [ 2294.242842] LustreError: 71098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.266846] LustreError: 71098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2297.091892] LustreError: 65373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440324 with bad export cookie 311369828511008563 [ 2297.096711] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2297.111294] LustreError: 65373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2297.485719] Lustre: server umount lustre-MDT0001 complete [ 2306.051568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2310.110922] Lustre: DEBUG MARKER: files compared the same [ 2321.612016] Lustre: server umount lustre-OST0000 complete [ 2337.947196] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2341.626711] Key type lgssc unregistered [ 2341.976982] LNet: 74374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2341.989118] LNetError: 74374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2342.005184] LNet: Removed LNI 192.168.204.121@tcp [ 2343.056312] Key type .llcrypt unregistered [ 2343.065709] Key type ._llcrypt unregistered [ 2372.986828] Key type ._llcrypt registered [ 2372.989085] Key type .llcrypt registered [ 2373.077830] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2373.949290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2373.994238] alg: No test for adler32 (adler32-zlib) [ 2375.074434] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2375.360490] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2377.088701] Key type lgssc registered [ 2378.351580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2388.436414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2388.493335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2390.207499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2395.366666] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug +malloc all [ 2399.713492] LustreError: 75788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.746302] LustreError: 75788:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2404.840186] LustreError: 75789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.444727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.861195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2410.452553] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug +malloc all [ 2419.314708] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2426.246447] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2433.083711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2433.446613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.483677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2440.687653] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug +malloc all [ 2449.231492] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2460.129370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2460.135932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2460.150760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2461.154985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2461.155547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2461.156216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2461.191715] Lustre: Skipped 1 previous similar message [ 2465.341569] Lustre: server umount lustre-OST0000 complete [ 2471.409265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.420525] Lustre: Skipped 1 previous similar message [ 2471.420733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2471.438956] Lustre: Skipped 1 previous similar message [ 2476.117288] Lustre: server umount lustre-MDT0000 complete [ 2476.514751] LustreError: 76793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.537501] LustreError: 76793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2480.534362] LustreError: 75769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440508 with bad export cookie 8201271517258665300 [ 2480.535052] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2480.553190] LustreError: 75769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2480.839742] Lustre: server umount lustre-MDT0001 complete [ 2495.509910] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2499.036794] Key type lgssc unregistered [ 2499.413917] LNet: 79334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2499.435160] LNetError: 79334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2499.464235] LNet: Removed LNI 192.168.204.121@tcp [ 2500.819107] Key type .llcrypt unregistered [ 2500.824847] Key type ._llcrypt unregistered [ 2529.306885] Key type ._llcrypt registered [ 2529.309796] Key type .llcrypt registered [ 2529.485466] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2530.750890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2530.980325] alg: No test for adler32 (adler32-zlib) [ 2532.068907] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2532.336541] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2534.096245] Key type lgssc registered [ 2535.331134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2544.926805] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2544.950548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2560.546341] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2565.664321] LustreError: 80722:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 4: rc = -110 [ 2592.480568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2598.891660] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2610.465783] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2620.373761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2620.557735] Lustre: *** cfs_fail_loc=706, val=0*** [ 2621.182103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2621.260894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2626.535036] LustreError: 80747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.558942] LustreError: 80747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2626.656803] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2627.557446] LustreError: 81808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.581197] LustreError: 81808:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2632.681875] LustreError: 81809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.704852] LustreError: 81809:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2636.985629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2637.377430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2643.364741] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2652.704550] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2659.004304] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2662.369301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2662.374314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2662.392383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2662.883808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2662.909574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2662.922218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2667.759851] Lustre: server umount lustre-OST0000 complete [ 2672.034867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.044819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2672.057887] Lustre: Skipped 1 previous similar message [ 2672.074843] Lustre: Skipped 1 previous similar message [ 2677.221620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2677.233264] Lustre: Skipped 1 previous similar message [ 2678.248765] Lustre: server umount lustre-MDT0000 complete [ 2681.739325] LustreError: 80730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440709 with bad export cookie 10574918079502466531 [ 2681.742096] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2681.756681] LustreError: 80730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2682.009304] Lustre: server umount lustre-MDT0001 complete [ 2694.047436] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2698.713321] Key type lgssc unregistered [ 2699.166181] LNet: 84337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2699.172676] LNetError: 84337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2699.186844] LNet: Removed LNI 192.168.204.121@tcp [ 2700.349868] Key type .llcrypt unregistered [ 2700.353797] Key type ._llcrypt unregistered [ 2730.158829] Key type ._llcrypt registered [ 2730.164201] Key type .llcrypt registered [ 2730.438902] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2731.842512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2731.899832] alg: No test for adler32 (adler32-zlib) [ 2732.992881] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2733.477763] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2735.328929] Key type lgssc registered [ 2737.009395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2746.539676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2746.556618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.445625] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2761.533733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.047626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2766.736822] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2767.330717] LustreError: 86439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.353345] LustreError: 86439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2771.425754] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.450404] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2776.537286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2776.559787] LustreError: 86442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2776.587303] LustreError: 86442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2777.062641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.180169] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.207125] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2784.534483] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2787.315113] LustreError: 86439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.336564] LustreError: 86439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2794.029947] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2794.064968] Lustre: MGS: Client 0dbb6784-dee5-4770-9a3a-0c44bbd062d5 (at 0@lo) reconnecting [ 2794.099336] LustreError: 86453:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2794.498837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2794.683795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2800.159235] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2814.948563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2814.951987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.965675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2814.968175] Lustre: Skipped 1 previous similar message [ 2814.980847] Lustre: Skipped 1 previous similar message [ 2819.643056] Lustre: server umount lustre-OST0000 complete [ 2825.185860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.186545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2825.216301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2825.230377] Lustre: Skipped 1 previous similar message [ 2830.307482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2830.317273] Lustre: Skipped 1 previous similar message [ 2830.776474] Lustre: server umount lustre-MDT0000 complete [ 2834.402144] LustreError: 86442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2834.421045] LustreError: 86442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2835.616721] Lustre: server umount lustre-MDT0001 complete [ 2839.948230] Lustre: server umount MGS complete [ 2853.963588] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:08:00 (1788440880) [ 2886.827418] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 2897.842612] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2937.890594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2943.934823] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2955.344055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.671675] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2955.776383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2955.916947] Lustre: lustre-MDT0001: new disk, initializing [ 2956.092583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2956.185282] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2956.199378] Lustre: 93611:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2956.215928] LustreError: 93611:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2961.258881] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2973.300406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2973.569203] Lustre: lustre-OST0000: new disk, initializing [ 2973.572823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2973.578627] Lustre: 94377:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2973.627942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2981.612479] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2982.503588] Lustre: 94717:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000003ce15527 x1875316059448576/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788441026 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2991.804767] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2998.914615] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2998.931780] Lustre: MGS: Client 696b8121-3328-4837-9681-34bb00941e77 (at 0@lo) reconnecting [ 2999.002797] Lustre: 92880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2999.018838] LustreError: 93597:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2999.031312] LustreError: 93597:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2999.070836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2999.167471] Lustre: lustre-MDT0000: new disk, initializing [ 2999.322878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2999.341362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2999.360171] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3000.418683] Lustre: 95447:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ae5c1f14 x1875316059455488/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788441044 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3004.209527] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3005.688120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3005.703153] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3005.782776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3009.551239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 3009.555787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3031.528262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3031.532713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3031.538860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3031.553769] Lustre: Skipped 2 previous similar messages [ 3036.649795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3036.666443] Lustre: Skipped 2 previous similar messages [ 3037.603921] Lustre: server umount lustre-OST0000 complete [ 3041.762425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3041.762961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.791019] Lustre: Skipped 1 previous similar message [ 3046.886457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3046.900110] Lustre: Skipped 3 previous similar messages [ 3048.016842] Lustre: server umount lustre-MDT0000 complete [ 3052.004649] LustreError: 93582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.031113] LustreError: 93582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3058.805948] Lustre: server umount lustre-MDT0001 complete [ 3063.774719] Lustre: server umount MGS complete [ 3077.757579] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:11:44 (1788441104) [ 3098.327516] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3101.822761] Key type lgssc unregistered [ 3102.270203] LNet: 97944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3102.285422] LNetError: 97944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3102.316695] LNet: Removed LNI 192.168.204.121@tcp [ 3103.364446] Key type .llcrypt unregistered [ 3103.369811] Key type ._llcrypt unregistered [ 3123.513520] Key type ._llcrypt registered [ 3123.516522] Key type .llcrypt registered [ 3123.642585] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3124.577335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3124.747456] alg: No test for adler32 (adler32-zlib) [ 3125.874418] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3126.161460] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3127.840219] Key type lgssc registered [ 3129.213740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3138.139569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3138.170778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3139.745811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3144.453810] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3150.316474] LustreError: 99163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3150.341785] LustreError: 99163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3153.468088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.894888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3158.620358] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3166.833932] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3174.067108] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3182.041692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3182.552488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3185.657629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3190.746598] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3201.106776] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3211.238717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3211.242837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3211.249238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3214.314913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3214.320855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.327816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3217.112661] Lustre: server umount lustre-OST0000 complete [ 3224.553134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3224.557394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3224.577973] Lustre: Skipped 1 previous similar message [ 3224.579972] Lustre: Skipped 1 previous similar message [ 3229.665678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3229.684806] Lustre: Skipped 2 previous similar messages [ 3234.788486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3234.796935] Lustre: Skipped 1 previous similar message [ 3235.300918] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3235.561099] Lustre: server umount lustre-MDT0000 complete [ 3239.910557] LustreError: 99164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3239.947471] LustreError: 99164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3240.603863] LustreError: 99143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441268 with bad export cookie 8591301665735922402 [ 3240.621303] LustreError: 99143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3240.621815] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3241.800800] Lustre: server umount lustre-MDT0001 complete [ 3256.498841] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3260.753888] Key type lgssc unregistered [ 3261.268646] LNet: 102699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3261.292188] LNetError: 102699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3261.336567] LNet: Removed LNI 192.168.204.121@tcp [ 3262.489178] Key type .llcrypt unregistered [ 3262.492707] Key type ._llcrypt unregistered [ 3282.252311] Key type ._llcrypt registered [ 3282.258145] Key type .llcrypt registered [ 3282.418708] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3283.567549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3283.770554] alg: No test for adler32 (adler32-zlib) [ 3285.146847] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3285.468697] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3287.200934] Key type lgssc registered [ 3288.819661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3296.239386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3296.249076] LustreError: 103783:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3296.255788] LustreError: 103784:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3296.261499] LustreError: 103783:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 3296.270670] LustreError: 103783:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3296.278682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3296.281132] LustreError: 103783:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3297.758705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3306.969693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3307.348630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3312.937379] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3320.198982] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing lsmod [ 3323.579578] LustreError: 104980:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3323.651209] LustreError: 104993:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3323.659363] LustreError: 104980:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4930 [ 3323.667818] LustreError: 104980:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3323.684354] LustreError: 104980:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3323.695860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3324.208956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3327.989245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3328.012938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3328.043333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3329.510709] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3329.515076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3329.531241] Lustre: Skipped 1 previous similar message [ 3329.539625] Lustre: Skipped 1 previous similar message [ 3331.627599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3331.635190] Lustre: Skipped 3 previous similar messages [ 3334.182995] Lustre: server umount lustre-MDT0000 complete [ 3336.674970] LustreError: 103823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3336.707795] LustreError: 103823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3339.062277] LustreError: 104263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441366 with bad export cookie 11942028269488391672 [ 3339.084805] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3339.091689] LustreError: 104263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3339.936314] LustreError: 104254:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e2249138 x1875316638242816/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3340.199173] Lustre: server umount lustre-MDT0001 complete [ 3355.262390] Lustre: server umount lustre-OST0000 complete [ 3365.757613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3366.367321] LustreError: 106056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.457069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3371.489450] LustreError: 106057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3371.925903] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3376.612855] LustreError: 106056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3381.733388] LustreError: 106057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.204416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3382.644827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3388.027779] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3397.443587] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3404.662666] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3413.655793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3414.154729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.652152] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3431.618591] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3436.670672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3447.270805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3447.276610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3447.278764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3447.308869] Lustre: Skipped 1 previous similar message [ 3451.088735] Lustre: server umount lustre-OST0000 complete [ 3457.515572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3457.521777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3457.539909] Lustre: Skipped 1 previous similar message [ 3457.569404] Lustre: Skipped 2 previous similar messages [ 3462.137705] Lustre: server umount lustre-MDT0000 complete [ 3462.625565] LustreError: 107348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3462.653913] LustreError: 107348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3466.782289] LustreError: 106036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441494 with bad export cookie 11942028269488392365 [ 3466.784230] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3466.794590] LustreError: 106036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3467.345077] Lustre: server umount lustre-MDT0001 complete [ 3481.887107] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3485.430980] Key type lgssc unregistered [ 3485.813355] LNet: 109616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3485.825883] LNetError: 109616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3485.857051] LNet: Removed LNI 192.168.204.121@tcp [ 3486.992247] Key type .llcrypt unregistered [ 3486.999209] Key type ._llcrypt unregistered [ 3515.838502] Key type ._llcrypt registered [ 3515.845471] Key type .llcrypt registered [ 3515.991229] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3516.973442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3517.055825] alg: No test for adler32 (adler32-zlib) [ 3518.401523] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3518.655642] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3520.392187] Key type lgssc registered [ 3521.555460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3530.150389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3530.168976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3531.835451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3537.799195] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3541.987082] LustreError: 111031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3542.005614] LustreError: 111031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3547.105915] LustreError: 111032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3548.011187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3548.576111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3553.978502] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3563.775118] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3571.445504] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3580.198158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3580.642995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3587.437500] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3596.204716] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3602.166531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3606.775909] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3611.386393] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3617.777649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3617.785779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3617.793929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3617.807667] Lustre: Skipped 1 previous similar message [ 3622.084863] Lustre: server umount lustre-OST0000 complete [ 3628.004630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3628.008411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3628.029456] Lustre: Skipped 1 previous similar message [ 3628.054478] Lustre: Skipped 2 previous similar messages [ 3632.606693] Lustre: server umount lustre-MDT0000 complete [ 3633.121939] LustreError: 112465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3633.160220] LustreError: 112465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3636.877721] LustreError: 111011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441664 with bad export cookie 10780681420912862643 [ 3636.898504] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3636.900811] LustreError: 111011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3637.330383] Lustre: server umount lustre-MDT0001 complete [ 3650.508307] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3653.667446] Key type lgssc unregistered [ 3653.948983] LNet: 114677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3653.956753] LNetError: 114677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3653.981767] LNet: Removed LNI 192.168.204.121@tcp [ 3655.104917] Key type .llcrypt unregistered [ 3655.109684] Key type ._llcrypt unregistered [ 3676.314415] Key type ._llcrypt registered [ 3676.316804] Key type .llcrypt registered [ 3676.448574] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3677.458403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3677.529978] alg: No test for adler32 (adler32-zlib) [ 3678.576830] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3678.807422] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3680.480284] Key type lgssc registered [ 3681.474694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3690.524191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3690.559948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3692.317692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3698.271435] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3701.731932] LustreError: 115894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3701.759295] LustreError: 115894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3706.858416] LustreError: 115895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3708.109508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.645467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3713.798834] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3721.933324] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3728.790113] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3736.118154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3736.409572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3737.489904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3743.670928] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3753.471159] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3763.171235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3763.176597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.186715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3764.199322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3764.210417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3764.210464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.228856] Lustre: Skipped 1 previous similar message [ 3768.624357] Lustre: server umount lustre-OST0000 complete [ 3774.435107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3774.459124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3774.464978] Lustre: Skipped 1 previous similar message [ 3778.744765] Lustre: server umount lustre-MDT0000 complete [ 3779.554993] LustreError: 115889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3779.582893] LustreError: 115889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3782.433825] LustreError: 115874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441810 with bad export cookie 7024833895413569477 [ 3782.440955] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3782.442907] LustreError: 115874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3782.759976] Lustre: server umount lustre-MDT0001 complete [ 3795.016258] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3798.214324] Key type lgssc unregistered [ 3798.518804] LNet: 119430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3798.527312] LNetError: 119430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3798.551743] LNet: Removed LNI 192.168.204.121@tcp [ 3799.482207] Key type .llcrypt unregistered [ 3799.485016] Key type ._llcrypt unregistered [ 3824.666872] Key type ._llcrypt registered [ 3824.668437] Key type .llcrypt registered [ 3824.786824] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3826.107693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3826.124661] alg: No test for adler32 (adler32-zlib) [ 3827.165460] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3827.387251] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3829.080192] Key type lgssc registered [ 3830.310586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3838.564626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3838.595822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3840.173681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3845.152760] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3849.700040] LustreError: 120844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3849.719854] LustreError: 120844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3854.028551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3854.509657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3859.718134] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3867.542872] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3874.770966] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3882.130727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3882.575521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3884.613737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3890.152758] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3899.525708] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3907.659918] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3911.507287] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3911.512935] Lustre: Skipped 2 previous similar messages [ 3914.854550] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3914.900536] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3916.752295] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3916.757790] Lustre: Skipped 2 previous similar messages [ 3920.318495] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3920.322419] Lustre: Skipped 1 previous similar message [ 3926.687243] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.21@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3926.700412] Lustre: Skipped 2 previous similar messages [ 3928.641667] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.21@tcp 192.168.0.[2,10]@tcp [ 3933.807850] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.21/24@tcp in log lustre-MDT0000 [ 3933.818275] Lustre: Skipped 2 previous similar messages [ 3936.195211] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.21/24@tcp [ 3936.205073] Lustre: Skipped 1 previous similar message [ 3945.955926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3945.957720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.971161] LustreError: Skipped 1 previous similar message [ 3945.974924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3945.981451] Lustre: Skipped 1 previous similar message [ 3946.002957] Lustre: Skipped 1 previous similar message [ 3949.657901] Lustre: server umount lustre-OST0000 complete [ 3956.202596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3956.213401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3956.227788] Lustre: Skipped 1 previous similar message [ 3956.243654] Lustre: Skipped 1 previous similar message [ 3960.807945] Lustre: server umount lustre-MDT0000 complete [ 3961.314102] LustreError: 120844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3961.345930] LustreError: 120844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3965.205628] LustreError: 123479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441992 with bad export cookie 18062651072465788888 [ 3965.218551] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3965.228216] LustreError: 123479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3965.646875] Lustre: server umount lustre-MDT0001 complete [ 3979.693115] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3982.556642] Key type lgssc unregistered [ 3982.902036] LNet: 125598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3982.913344] LNetError: 125598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3982.938668] LNet: Removed LNI 192.168.204.121@tcp [ 3983.758244] Key type .llcrypt unregistered [ 3983.761527] Key type ._llcrypt unregistered [ 4016.745428] Key type ._llcrypt registered [ 4016.748201] Key type .llcrypt registered [ 4016.881903] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 4017.585134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4017.701611] alg: No test for adler32 (adler32-zlib) [ 4019.117919] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4019.498955] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 4021.208309] Key type lgssc registered [ 4022.694596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4032.214434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4032.247261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4033.814161] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4033.852779] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.21/24@tcp [ 4033.886279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4038.890106] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4044.260930] LustreError: 127154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4044.271077] LustreError: 127154:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4048.374955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4048.695988] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4048.710978] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.21/24@tcp [ 4048.745856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4054.120289] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4061.737937] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4068.284424] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4075.889405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4076.351491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4078.420741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 4084.573073] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4093.323878] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4107.793945] Lustre: server umount lustre-OST0000 complete [ 4114.408135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.430580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4114.442581] Lustre: Skipped 1 previous similar message [ 4119.522178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4119.536635] Lustre: Skipped 1 previous similar message [ 4122.680679] Lustre: server umount lustre-MDT0000 complete [ 4124.640189] Lustre: 126463:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442136/real 1788442136] req@00000000e4c1b13a x1875317407904896/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788442152 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4124.648599] LustreError: 128381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4124.673679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4124.692830] LustreError: 128381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4124.706346] Lustre: Skipped 1 previous similar message [ 4126.706688] LustreError: 127135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442154 with bad export cookie 16850542404740998543 [ 4126.707465] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4126.717738] LustreError: 127135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4127.090651] Lustre: server umount lustre-MDT0001 complete [ 4140.431242] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 4143.528786] Key type lgssc unregistered [ 4143.822496] LNet: 130746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4143.830555] LNetError: 130746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4143.855244] LNet: Removed LNI 192.168.204.121@tcp [ 4144.882830] Key type .llcrypt unregistered [ 4144.886114] Key type ._llcrypt unregistered [ 4173.647740] Key type ._llcrypt registered [ 4173.650394] Key type .llcrypt registered [ 4173.756372] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 4191.276175] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 4192.712431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4192.732600] alg: No test for adler32 (adler32-zlib) [ 4194.296367] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4194.630708] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 4196.384219] Key type lgssc registered [ 4197.796714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4237.390586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4237.418984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.690964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4238.723714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4238.828549] Lustre: lustre-MDT0000: new disk, initializing [ 4238.937860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4238.959244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4244.159447] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4255.933801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4256.071312] Lustre: 134145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4256.151513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4256.158934] Lustre: Skipped 1 previous similar message [ 4256.252588] Lustre: lustre-MDT0001: new disk, initializing [ 4256.451336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4256.518386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4256.536804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4262.683285] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4274.563378] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4281.974888] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4290.127096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4290.327770] Lustre: lustre-OST0000: new disk, initializing [ 4290.332170] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4290.337331] Lustre: 136112:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4290.445647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4296.799794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4296.816162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4297.017150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4297.216724] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4307.289599] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4312.524410] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4312.738096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4318.216716] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4318.434493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4322.795988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4322.816475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4322.853220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4326.930677] Lustre: server umount lustre-OST0000 complete [ 4333.029808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4333.031504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4333.039981] LustreError: Skipped 1 previous similar message [ 4333.042172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4333.058494] Lustre: Skipped 2 previous similar messages [ 4333.075368] Lustre: Skipped 2 previous similar messages [ 4338.146862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4338.159137] Lustre: Skipped 1 previous similar message [ 4343.274610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4343.279736] Lustre: Skipped 1 previous similar message [ 4346.340219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4346.550723] Lustre: server umount lustre-MDT0000 complete [ 4350.178680] LustreError: 134136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442377 with bad export cookie 10648587315674614338 [ 4350.190558] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4350.597156] Lustre: server umount lustre-MDT0001 complete [ 4359.899312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4360.317616] LustreError: 138329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4360.348853] LustreError: 138329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4360.444339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.244126] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4365.798161] LustreError: 138330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4370.914843] LustreError: 138329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4374.407153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4374.833509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4380.383728] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4389.873869] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4397.045971] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4404.489851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4404.941589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4411.890622] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4420.606862] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4425.848668] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4426.089579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4431.095216] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4431.365309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4440.555671] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4440.692992] Lustre: lustre-OST0001: new disk, initializing [ 4440.695962] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4440.709408] Lustre: 141700:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4440.724936] Lustre: lustre-OST0001: Not available for connect from 192.168.204.21@tcp (not set up) [ 4440.791379] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4447.616740] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4448.371564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4448.382209] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4448.510745] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4457.246388] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4457.433716] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4462.671596] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4462.937227] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4469.534523] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f7687553000.ost_server_uuid 50 [ 4471.341928] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f7687553000.ost_server_uuid in IDLE state after 0 sec [ 4482.539615] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.550833] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4482.553413] Lustre: Skipped 1 previous similar message [ 4482.569765] Lustre: Skipped 1 previous similar message [ 4487.114535] Lustre: server umount lustre-OST0001 complete [ 4487.672576] LustreError: 140150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4487.725490] LustreError: 140150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4492.263063] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4492.269937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4492.773365] LustreError: 140151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4492.794757] LustreError: 140151:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4497.511761] Lustre: server umount lustre-OST0000 complete [ 4503.009494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4503.012464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4503.024332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4503.049130] Lustre: Skipped 2 previous similar messages [ 4503.071625] Lustre: Skipped 3 previous similar messages [ 4508.004457] Lustre: server umount lustre-MDT0000 complete [ 4508.129303] LustreError: 138330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4508.157172] LustreError: 138330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4512.295624] LustreError: 140171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442539 with bad export cookie 10648587315674615227 [ 4512.298654] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4512.311437] LustreError: 140171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4512.924393] Lustre: server umount lustre-MDT0001 complete [ 4536.534637] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 4539.660997] Key type lgssc unregistered [ 4539.988853] LNet: 144706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4539.996507] LNetError: 144706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4540.023121] LNet: Removed LNI 192.168.204.121@tcp [ 4541.177454] Key type .llcrypt unregistered [ 4541.179053] Key type ._llcrypt unregistered [ 4584.413656] Key type ._llcrypt registered [ 4584.415785] Key type .llcrypt registered [ 4584.544791] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 4601.385219] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 4602.909213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4602.927944] alg: No test for adler32 (adler32-zlib) [ 4604.067267] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4604.383361] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 4606.056182] Key type lgssc registered [ 4607.318579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4645.466972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4645.525246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4646.843335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4646.887652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4647.010622] Lustre: lustre-MDT0000: new disk, initializing [ 4647.102812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.120657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4651.737511] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4662.916529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4663.095329] Lustre: 148856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4663.180534] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4663.186501] Lustre: Skipped 1 previous similar message [ 4663.250979] Lustre: lustre-MDT0001: new disk, initializing [ 4663.397562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4663.466381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4663.494610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4669.412196] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4680.926486] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4688.421339] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4696.136969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4696.411967] Lustre: lustre-OST0000: new disk, initializing [ 4696.420565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4696.439590] Lustre: 150826:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4696.622608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4703.523464] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4705.365245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4705.379561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4705.498791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4714.198958] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4719.479292] Lustre: Failing over lustre-OST0000 [ 4721.121709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4721.131024] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4721.134192] LustreError: Skipped 1 previous similar message [ 4721.136792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4721.161384] Lustre: Skipped 1 previous similar message [ 4721.185133] Lustre: Skipped 1 previous similar message [ 4721.638863] Lustre: server umount lustre-OST0000 complete [ 4741.961891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4742.532986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4742.565595] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4747.497340] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4749.150198] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4754.232697] Lustre: Failing over lustre-MDT0000 [ 4754.250088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4754.258884] Lustre: Skipped 1 previous similar message [ 4754.264639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4754.275656] Lustre: Skipped 2 previous similar messages [ 4754.723811] Lustre: server umount lustre-MDT0000 complete [ 4762.849154] LustreError: 148863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4762.865994] LustreError: 148863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4765.683021] LustreError: 148869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4765.707561] LustreError: 148869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4767.974531] LustreError: 151472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.786051] LustreError: 152737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.799370] LustreError: 152737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4775.635672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4775.762527] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4775.909262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 4775.915294] Lustre: Skipped 2 previous similar messages [ 4776.160961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4776.241296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4776.335597] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4778.218532] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4781.574990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4781.636295] Lustre: lustre-OST0000: Recovery over after 0:34, of 3 clients 3 recovered and 0 were evicted. [ 4781.730367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4782.338296] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4791.778777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4791.780171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4791.787783] LustreError: Skipped 1 previous similar message [ 4791.796695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4791.843693] Lustre: Skipped 2 previous similar messages [ 4796.900968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4796.913090] Lustre: Skipped 1 previous similar message [ 4797.665760] Lustre: server umount lustre-OST0000 complete [ 4802.529496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4802.537714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4807.138693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4807.139489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4807.163673] Lustre: Skipped 3 previous similar messages [ 4808.792848] Lustre: server umount lustre-MDT0000 complete [ 4812.271121] LustreError: 148869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4812.289105] LustreError: 148869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4813.025387] LustreError: 148848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442840 with bad export cookie 6833771364091136711 [ 4813.035027] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4813.039373] LustreError: 148848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4813.509049] Lustre: server umount lustre-MDT0001 complete [ 4825.924911] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 4829.333209] Key type lgssc unregistered [ 4829.701563] LNet: 154483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4829.726613] LNetError: 154483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4830.766339] LNet: Removed LNI 192.168.204.121@tcp [ 4832.062281] Key type .llcrypt unregistered [ 4832.066048] Key type ._llcrypt unregistered [ 4871.726806] Key type ._llcrypt registered [ 4871.729627] Key type .llcrypt registered [ 4871.814676] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 4872.864063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4872.907166] alg: No test for adler32 (adler32-zlib) [ 4874.255430] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4874.607374] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 4876.417883] Key type lgssc registered [ 4877.784913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4885.282839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4885.308647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4886.845926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4891.659442] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4897.250469] LustreError: 156000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4897.281974] LustreError: 156000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4902.381390] LustreError: 156001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4903.326435] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4907.500057] LustreError: 156000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4907.979768] LustreError: 156029:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4907.998765] LustreError: 156029:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4908.010382] LustreError: 156029:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4908.497328] Lustre: server umount lustre-MDT0000 complete [ 4928.512654] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 4939.884499] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 4982.993943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4983.275156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4983.326503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4983.459167] Lustre: lustre-MDT0000: new disk, initializing [ 4983.560493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4983.578486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4988.917570] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5000.913898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5001.093142] Lustre: 159336:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5001.194875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5001.203000] Lustre: Skipped 1 previous similar message [ 5001.310293] Lustre: lustre-MDT0001: new disk, initializing [ 5001.453097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5001.524313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5001.540939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5006.473434] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5017.524106] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5024.233415] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5032.005399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5032.278814] Lustre: lustre-OST0000: new disk, initializing [ 5032.282434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5032.290731] Lustre: 161307:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5032.370945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5038.702659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5038.731147] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5038.897597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5039.624691] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5052.408239] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5057.689841] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5059.394533] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:44:46 (1788443086) [ 5365.218318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5365.231576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5365.242181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5366.753494] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5366.754748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5366.789826] Lustre: Skipped 1 previous similar message [ 5366.806329] Lustre: Skipped 2 previous similar messages [ 5369.520384] Lustre: server umount lustre-MDT0000 complete [ 5371.872969] LustreError: 160520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5371.902212] LustreError: 160520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5373.582521] LustreError: 159328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443401 with bad export cookie 11289474501222871793 [ 5373.591849] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5373.598238] LustreError: 159328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5376.993083] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5376.993733] LustreError: 161584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5377.018359] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5379.043872] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5379.985748] Lustre: server umount lustre-MDT0001 complete [ 5390.329302] Lustre: server umount lustre-OST0000 complete [ 5401.214210] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 5413.983604] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 5458.155637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5458.452661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5458.553721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5458.691283] Lustre: lustre-MDT0000: new disk, initializing [ 5458.803114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5458.823536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5464.581864] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5477.146125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5477.391635] Lustre: 165983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5477.523446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5477.537885] Lustre: Skipped 1 previous similar message [ 5477.643314] Lustre: lustre-MDT0001: new disk, initializing [ 5477.791861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5477.844373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5477.857793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5483.259973] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5494.729549] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5500.988351] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5509.044122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5509.436799] Lustre: lustre-OST0000: new disk, initializing [ 5509.447133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5509.466517] Lustre: 167951:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5509.572575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5514.842957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5514.861584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5514.956508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5516.981113] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5527.719314] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5532.960669] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5533.235095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5538.772421] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5539.020635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5545.960440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5545.965309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.969549] LustreError: Skipped 1 previous similar message [ 5545.988986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5545.997527] Lustre: Skipped 1 previous similar message [ 5547.559349] Lustre: server umount lustre-OST0000 complete [ 5554.662365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5554.676207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5554.697982] Lustre: Skipped 1 previous similar message [ 5554.707470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5557.898798] Lustre: server umount lustre-MDT0000 complete [ 5561.315932] LustreError: 165993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5561.345264] LustreError: 165993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5562.031178] LustreError: 165973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443589 with bad export cookie 11289474501223030616 [ 5562.033716] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5562.043587] LustreError: 165973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5562.525220] Lustre: server umount lustre-MDT0001 complete [ 5572.022575] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:53:18 (1788443598) [ 5589.470487] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 5602.047246] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 5638.876669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.175383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5639.219320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5639.357534] Lustre: lustre-MDT0000: new disk, initializing [ 5639.424948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5639.449492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5644.367925] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5655.331482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5655.457687] Lustre: 172712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5655.683486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5660.815473] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5672.308403] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5680.142446] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5687.579869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5688.082750] Lustre: lustre-OST0000: new disk, initializing [ 5688.087405] Lustre: Skipped 1 previous similar message [ 5688.100736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5688.114075] Lustre: Skipped 2 previous similar messages [ 5688.126444] Lustre: 174690:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5695.373666] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5697.137623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5697.151602] Lustre: Skipped 1 previous similar message [ 5697.159860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5697.260092] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5705.461712] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5717.989838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5717.993291] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5718.008768] Lustre: Skipped 1 previous similar message [ 5718.014001] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5718.034798] Lustre: Skipped 3 previous similar messages [ 5720.077387] Lustre: server umount lustre-OST0000 complete [ 5727.201981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5730.353601] Lustre: server umount lustre-MDT0000 complete [ 5733.345603] LustreError: 174083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5733.356967] LustreError: 174083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5734.258337] LustreError: 172703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443761 with bad export cookie 11289474501223031498 [ 5734.270652] LustreError: 172703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5734.273323] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5734.613675] Lustre: server umount lustre-MDT0001 complete [ 5743.612770] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:56:10 (1788443770) [ 5760.591579] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 5773.068582] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 5813.194590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5813.575411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5813.625724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5813.756593] Lustre: lustre-MDT0000: new disk, initializing [ 5813.857840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5813.865658] Lustre: Skipped 2 previous similar messages [ 5813.882812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5819.347081] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5831.056324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5831.178300] Lustre: 179016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5831.484028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5837.253771] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5847.689631] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5854.075938] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5863.866466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5864.215600] Lustre: 180995:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5870.983556] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5873.773025] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5873.914459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5882.517858] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5894.629326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5894.635671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5894.646158] LustreError: Skipped 1 previous similar message [ 5894.648417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5894.648424] Lustre: Skipped 3 previous similar messages [ 5894.676519] Lustre: Skipped 4 previous similar messages [ 5897.681916] Lustre: server umount lustre-OST0000 complete [ 5908.447266] Lustre: server umount lustre-MDT0000 complete [ 5909.995130] LustreError: 179028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5910.021690] LustreError: 179028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5913.106162] LustreError: 179008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443940 with bad export cookie 11289474501223032590 [ 5913.111547] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5913.118289] LustreError: 179008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5913.467257] Lustre: server umount lustre-MDT0001 complete [ 5926.766576] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 5930.244655] Key type lgssc unregistered [ 5930.688234] LNet: 182850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5930.706167] LNetError: 182850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5930.722775] LNet: Removed LNI 192.168.204.121@tcp [ 5931.698697] Key type .llcrypt unregistered [ 5931.700912] Key type ._llcrypt unregistered [ 5961.834756] Key type ._llcrypt registered [ 5961.837839] Key type .llcrypt registered [ 5962.039458] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 5963.930312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5963.963192] alg: No test for adler32 (adler32-zlib) [ 5965.234612] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5965.594317] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 5967.281304] Key type lgssc registered [ 5968.596392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5976.416428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5976.441229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.864500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5982.863225] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5987.364508] LustreError: 184263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5987.389017] LustreError: 184263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5992.142121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5992.417917] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5992.688814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5997.768369] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6005.516743] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6012.685914] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6019.573535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6020.075540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6023.189672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6027.180952] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6034.512645] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6058.981954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6058.994804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6059.007084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6063.074102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6063.085661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6063.088325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6063.129016] Lustre: Skipped 1 previous similar message [ 6064.489532] Lustre: server umount lustre-OST0000 complete [ 6069.730987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6069.741159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6069.754870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6073.319369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6074.707654] Lustre: server umount lustre-MDT0000 complete [ 6078.442323] LustreError: 184264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6078.454257] LustreError: 184264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6078.908979] LustreError: 184244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444106 with bad export cookie 7076792276084536244 [ 6078.918483] LustreError: 184244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6078.918922] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6079.183625] Lustre: server umount lustre-MDT0001 complete [ 6094.914262] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 6098.101195] Key type lgssc unregistered [ 6098.525490] LNet: 187832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6098.535760] LNetError: 187832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6098.580492] LNet: Removed LNI 192.168.204.121@tcp [ 6099.511317] Key type .llcrypt unregistered [ 6099.513647] Key type ._llcrypt unregistered [ 6124.671494] Key type ._llcrypt registered [ 6124.674236] Key type .llcrypt registered [ 6124.794693] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 6126.226053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6126.247862] alg: No test for adler32 (adler32-zlib) [ 6127.378775] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6127.672839] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 6129.392320] Key type lgssc registered [ 6130.854195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6138.938645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6138.969436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6140.358814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6144.497886] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6150.626872] LustreError: 189250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6150.633609] LustreError: 189250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6153.430241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6153.781138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6159.698987] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6168.573582] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6175.372537] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6182.296067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6182.616439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6185.720163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6188.995173] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6196.687771] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6204.385343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6204.390689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6204.407516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6204.418784] Lustre: Skipped 1 previous similar message [ 6209.510554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6209.532946] Lustre: Skipped 3 previous similar messages [ 6214.641308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6214.654572] Lustre: Skipped 2 previous similar messages [ 6218.208310] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6218.532167] Lustre: server umount lustre-OST0000 complete [ 6227.538981] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9f7687c1a800.ost_server_uuid 50 [ 6230.270883] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6249.448941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6249.472356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6249.484941] Lustre: Skipped 1 previous similar message [ 6249.518654] Lustre: Skipped 2 previous similar messages [ 6252.931083] Lustre: server umount lustre-MDT0000 complete [ 6254.563493] LustreError: 189247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6254.586063] LustreError: 189247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6257.180719] LustreError: 189230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444284 with bad export cookie 5610351637866826871 [ 6257.192881] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6257.193983] LustreError: 189230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6257.550400] Lustre: server umount lustre-MDT0001 complete [ 6267.775499] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:04:54 (1788444294) [ 6274.984259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6275.375767] LustreError: 193167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6275.425126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6280.676541] LustreError: 193168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6280.713978] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6285.797226] LustreError: 193167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6290.610457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6290.915574] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6291.174703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6296.062215] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6304.966766] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6311.524758] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6319.058922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6319.708951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6320.779844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6326.860142] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6337.536564] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6348.591331] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6348.865469] Lustre: lustre-OST0001: new disk, initializing [ 6348.871328] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6348.877715] Lustre: 196010:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6349.025148] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6357.056164] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6357.781826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6357.801321] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6357.952647] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6373.511529] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6377.442272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6377.448598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6377.453592] LustreError: Skipped 1 previous similar message [ 6377.456054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6377.470811] Lustre: Skipped 1 previous similar message [ 6377.495228] Lustre: Skipped 1 previous similar message [ 6383.048459] Lustre: server umount lustre-OST0000 complete [ 6383.600084] LustreError: 194989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6383.634896] LustreError: 194989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6391.124748] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6391.382423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6397.505741] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6397.799326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6399.727084] LustreError: 194990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6399.757131] LustreError: 194990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6414.307044] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6414.317805] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.323579] LustreError: Skipped 1 previous similar message [ 6414.349050] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6414.364943] Lustre: Skipped 1 previous similar message [ 6414.389804] Lustre: Skipped 4 previous similar messages [ 6419.600929] Lustre: server umount lustre-OST0001 complete [ 6424.546707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6424.554199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6424.561130] Lustre: Skipped 1 previous similar message [ 6431.140899] Lustre: server umount lustre-MDT0000 complete [ 6434.786552] LustreError: 194830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6434.795843] LustreError: 194830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6435.281187] LustreError: 193148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444462 with bad export cookie 5610351637866827942 [ 6435.282274] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6435.294107] LustreError: 193148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6435.615586] Lustre: server umount lustre-MDT0001 complete [ 6464.642979] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:08:10 (1788444490) [ 6472.813709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6473.124652] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6473.149466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6473.356293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6477.588734] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6486.110853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6486.231892] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6486.294760] Lustre: 199455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6491.511652] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6500.621700] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6507.686606] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6516.075494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6516.375960] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6524.161196] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6526.668512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6533.769861] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6542.518290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6542.632279] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6542.892285] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6542.904453] Lustre: Skipped 2 previous similar messages [ 6549.794983] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6552.388271] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6559.507649] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6565.644889] Lustre: lustre-OST0000: Not available for connect from 192.168.204.21@tcp (stopping) [ 6565.661976] Lustre: Skipped 4 previous similar messages [ 6567.911253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6567.920221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6577.120142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6577.250764] Lustre: server umount lustre-OST0000 complete [ 6578.152185] LustreError: 201305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6578.182247] LustreError: 201305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6598.628541] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6598.635526] LustreError: Skipped 1 previous similar message [ 6598.642900] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6598.675715] Lustre: Skipped 1 previous similar message [ 6600.539671] Lustre: server umount lustre-OST0001 complete [ 6611.697566] Lustre: server umount lustre-MDT0000 complete [ 6615.762598] LustreError: 202343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444643 with bad export cookie 5610351637866829349 [ 6615.768408] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6615.780996] LustreError: 202343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6616.308663] Lustre: server umount lustre-MDT0001 complete [ 6644.521813] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:11:10 (1788444670) [ 6669.042616] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 6680.930584] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 6721.587376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6721.857622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6721.923350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6722.058850] Lustre: lustre-MDT0000: new disk, initializing [ 6722.168287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6722.189691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6727.548223] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6739.534508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6739.650095] Lustre: 207557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6739.759566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6739.763856] Lustre: Skipped 1 previous similar message [ 6739.894180] Lustre: lustre-MDT0001: new disk, initializing [ 6740.105263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6740.115280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6745.734115] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6757.530552] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6764.743288] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6771.694338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6771.946688] Lustre: lustre-OST0000: new disk, initializing [ 6771.955120] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6771.960915] Lustre: 209528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6778.413852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6778.421261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6778.512251] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6778.754291] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6789.189689] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6795.506978] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6795.896414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6802.407815] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6802.738352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6809.570580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6809.597206] LustreError: Skipped 1 previous similar message [ 6809.609287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6809.629591] Lustre: Skipped 3 previous similar messages [ 6809.639798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6809.651702] Lustre: Skipped 10 previous similar messages [ 6819.297273] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6819.746686] Lustre: server umount lustre-OST0000 complete [ 6831.758199] Lustre: server umount lustre-MDT0000 complete [ 6835.179029] LustreError: 208636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6835.216387] LustreError: 208636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 6835.691696] LustreError: 207548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444863 with bad export cookie 5610351637866831141 [ 6835.701674] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6836.034201] Lustre: server umount lustre-MDT0001 complete [ 6845.205755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6851.058072] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6860.795255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6866.134735] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6875.283547] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6883.708447] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6893.183445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6900.942052] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6912.062627] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6918.156833] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6918.528949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6924.150095] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6924.456466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6926.817850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6926.834313] LustreError: Skipped 2 previous similar messages [ 6926.838227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6926.850234] Lustre: Skipped 3 previous similar messages [ 6932.855419] Lustre: server umount lustre-OST0000 complete [ 6943.068655] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6950.341198] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6980.315062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6980.596319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6980.599703] Lustre: Skipped 5 previous similar messages [ 6981.626925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6981.661188] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6981.681875] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6987.503618] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 6996.508321] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6999.421170] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7007.205126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7007.212825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7007.218616] LustreError: Skipped 1 previous similar message [ 7007.237847] Lustre: Skipped 11 previous similar messages [ 7012.102326] Lustre: server umount lustre-OST0000 complete [ 7022.352638] Lustre: server umount lustre-MDT0000 complete [ 7026.673172] LustreError: 211725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445054 with bad export cookie 5610351637866832023 [ 7026.676559] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7026.703399] LustreError: 211725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7027.099565] Lustre: server umount lustre-MDT0001 complete [ 7037.612546] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:17:44 (1788445064) [ 7046.580426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7052.300575] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7062.301915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7068.238774] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7077.663166] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7084.747162] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7092.573330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7100.599760] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7109.350199] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7114.819683] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7117.294579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7122.739376] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7123.066772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7130.996803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7131.269231] Lustre: lustre-OST0001: new disk, initializing [ 7131.273588] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7131.279571] Lustre: 221104:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7137.955252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7137.973675] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7138.131260] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7138.437423] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7149.581356] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7154.385757] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7154.624797] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7158.822386] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7159.069936] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7163.880710] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7163.887736] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7163.897827] Lustre: Skipped 5 previous similar messages [ 7163.909644] LustreError: Skipped 1 previous similar message [ 7175.146208] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7175.424840] Lustre: server umount lustre-OST0001 complete [ 7179.238108] LustreError: 221388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7179.245311] LustreError: 221388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7182.987452] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7183.331747] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7188.968377] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7189.266885] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7219.753813] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7225.358250] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7225.375840] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7225.377412] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7225.397837] Lustre: Skipped 2 previous similar messages [ 7227.457708] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7235.988119] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7238.031735] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7246.304291] Lustre: server umount lustre-OST0001 complete [ 7265.254078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7265.260761] Lustre: Skipped 12 previous similar messages [ 7270.006793] Lustre: server umount lustre-OST0000 complete [ 7288.288165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7288.522403] Lustre: server umount lustre-MDT0000 complete [ 7292.583831] LustreError: 217698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445320 with bad export cookie 5610351637866832989 [ 7292.588737] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7292.605110] LustreError: 217698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7299.116365] Lustre: server umount lustre-MDT0001 complete [ 7327.648822] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:22:34 (1788445354) [ 7335.989064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7336.257290] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7336.293868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7341.958815] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7350.888934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7350.987192] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7351.023978] Lustre: 226355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7356.494680] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7366.905048] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7374.427733] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7383.725940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7384.122505] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7391.384567] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7400.352590] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7410.901981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7410.991155] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7418.606285] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7430.566767] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7433.175736] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7439.086760] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7439.291732] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7446.619568] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7446.963855] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7453.835742] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f7685366000.ost_server_uuid 50 [ 7455.588916] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f7685366000.ost_server_uuid in IDLE state after 0 sec [ 7456.637619] Lustre: Permanently deactivating lustre-OST0001 [ 7456.674408] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7461.175732] Lustre: Permanently reactivating lustre-OST0001 [ 7461.218439] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7461.221136] Lustre: Skipped 2 previous similar messages [ 7465.953781] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7465.980374] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7465.983857] LustreError: Skipped 5 previous similar messages [ 7466.019843] Lustre: Skipped 8 previous similar messages [ 7480.289384] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7480.430547] Lustre: server umount lustre-OST0001 complete [ 7490.922987] Lustre: server umount lustre-OST0000 complete [ 7501.208538] Lustre: server umount lustre-MDT0000 complete [ 7505.199979] LustreError: 226348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445532 with bad export cookie 5610351637866834277 [ 7505.213931] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7505.214155] LustreError: 226348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7505.482429] Lustre: server umount lustre-MDT0001 complete [ 7533.753928] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:26:00 (1788445560) [ 7542.537338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7542.944787] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7542.984896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7542.992108] Lustre: Skipped 2 previous similar messages [ 7543.139215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7543.147217] Lustre: Skipped 9 previous similar messages [ 7548.078478] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7557.832295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7558.025276] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7558.066582] Lustre: 233055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7563.335744] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7571.415246] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7578.554893] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7585.723684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7585.989907] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7586.019071] Lustre: Permanently deactivating lustre-OST0000 [ 7586.039856] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7593.350482] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7594.067524] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7594.096300] LustreError: 235308:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 7594.113068] LustreError: 235308:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 7604.132805] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7614.189297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7614.233525] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7622.065481] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7622.905508] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7631.448653] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7633.834370] Lustre: Permanently reactivating lustre-OST0000 [ 7633.865389] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7633.876096] Lustre: Skipped 2 previous similar messages [ 7643.850064] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7643.852857] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 50 seconds. I think it's dead, and I am evicting it. exp 00000000473bdc62, cur 1788445671 deadline 1788445651 last 1788445621 [ 7643.858470] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7643.887155] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7643.957815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7657.463482] Lustre: server umount lustre-OST0001 complete [ 7668.805750] Lustre: server umount lustre-OST0000 complete [ 7688.160913] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7688.500870] Lustre: server umount lustre-MDT0000 complete [ 7692.802815] LustreError: 233046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445720 with bad export cookie 5610351637866835922 [ 7692.823711] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7692.824441] LustreError: 233046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7693.247511] Lustre: server umount lustre-MDT0001 complete [ 7706.742453] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 7710.135702] Key type lgssc unregistered [ 7710.615854] LNet: 237947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7710.629943] LNetError: 237947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7710.661892] LNet: Removed LNI 192.168.204.121@tcp [ 7711.896133] Key type .llcrypt unregistered [ 7711.900120] Key type ._llcrypt unregistered [ 7740.904487] Key type ._llcrypt registered [ 7740.907136] Key type .llcrypt registered [ 7740.995921] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 7742.459749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7742.584312] alg: No test for adler32 (adler32-zlib) [ 7743.703481] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7743.958663] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 7745.673175] Key type lgssc registered [ 7746.943856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7759.030696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7759.070573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7760.552955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7765.482703] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7771.108554] LustreError: 239411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7771.121857] LustreError: 239411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7774.769382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7774.908847] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7774.941470] Lustre: 239401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7774.961337] Lustre: Permanently deactivating lustre-MDT0001 [ 7774.996990] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7775.442441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7780.566406] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7781.407736] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7789.366722] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7796.637495] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7805.294038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7805.478477] LustreError: 241253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7805.760814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7811.049468] LustreError: 241255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7811.060228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7811.077398] LustreError: 241255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7813.913143] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7816.168288] LustreError: 241255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7821.313164] LustreError: 241256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7821.345164] LustreError: 241256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7824.525180] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7831.533669] LustreError: 241485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7831.552768] LustreError: 241485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7832.996660] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7833.245869] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7838.731981] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7838.752199] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7840.171608] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7848.657400] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7851.845918] Lustre: Permanently reactivating lustre-MDT0001 [ 7851.876204] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7851.893237] Lustre: Skipped 2 previous similar messages [ 7854.787660] Lustre: lustre-MDT0001: Client c3268758-2604-4e16-b0a4-3948f3b25676 (at 192.168.204.21@tcp) reconnecting [ 7861.354797] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7862.395482] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7862.420632] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7862.435487] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7862.472327] LustreError: 243171:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7862.475156] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7862.485747] LustreError: 243171:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7862.485924] LustreError: 243171:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.121@tcp: cfg command failed: rc = -108 [ 7862.486193] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7862.486193] [ 7862.487164] LustreError: 239403:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7863.362643] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7869.437303] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7869.707753] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7875.933408] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7876.298115] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7878.997957] Lustre: Permanently deactivating lustre-MDT0001 [ 7879.009493] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7879.016259] Lustre: Skipped 2 previous similar messages [ 7886.499939] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7892.963838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7892.973044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7892.986197] LustreError: 239412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7893.046573] Lustre: server umount lustre-MDT0000 complete [ 7895.032371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7895.075425] Lustre: Skipped 1 previous similar message [ 7897.697142] LustreError: 239393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445925 with bad export cookie 3574436305225682453 [ 7897.704384] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7897.706649] LustreError: 239393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7897.934717] Lustre: server umount lustre-MDT0001 complete [ 7902.765157] Lustre: server umount lustre-OST0000 complete [ 7918.403645] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:32:24 (1788445944) [ 7932.018313] Lustre: server umount lustre-OST0001 complete [ 7939.346339] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 7949.008769] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 7986.956662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7987.250906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7987.268506] Lustre: Skipped 2 previous similar messages [ 7987.294224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7987.365979] Lustre: lustre-MDT0000: new disk, initializing [ 7987.439074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7987.452312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7992.196648] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8003.509371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8003.631756] Lustre: 247674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8003.693522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8003.702405] Lustre: Skipped 1 previous similar message [ 8003.821440] Lustre: lustre-MDT0001: new disk, initializing [ 8003.924875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8003.983206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8003.991768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8008.695936] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8019.254840] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8026.107762] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8037.450727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8038.081429] Lustre: lustre-OST0000: new disk, initializing [ 8038.092231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8038.107946] Lustre: 249645:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8038.348840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8047.176099] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8048.238465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8048.256480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8048.410932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8058.319711] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8060.514073] LustreError: 247682:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8070.274577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8070.483670] Lustre: lustre-OST0001: new disk, initializing [ 8070.486790] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8070.492328] Lustre: 250761:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8070.609236] LustreError: 247682:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8070.636525] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8070.668025] LustreError: 249184:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8076.473185] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8078.407830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8078.421326] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8078.534910] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8080.744403] LustreError: 249184:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8085.553234] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8087.097687] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8091.432113] Lustre: lustre-OST0001: Not available for connect from 192.168.204.21@tcp (stopping) [ 8094.178457] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8094.183496] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8094.203054] LustreError: Skipped 1 previous similar message [ 8094.221554] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8094.243318] Lustre: Skipped 2 previous similar messages [ 8095.856427] Lustre: server umount lustre-OST0001 complete [ 8096.515397] LustreError: 249627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8096.552445] LustreError: 249627:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8114.663641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.686920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8114.693556] Lustre: Skipped 2 previous similar messages [ 8119.784255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8119.797049] Lustre: Skipped 1 previous similar message [ 8120.290511] Lustre: server umount lustre-OST0000 complete [ 8124.897903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8124.918248] Lustre: Skipped 1 previous similar message [ 8124.931595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8124.939425] Lustre: Skipped 1 previous similar message [ 8130.401670] Lustre: server umount lustre-MDT0000 complete [ 8133.929238] LustreError: 249644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446161 with bad export cookie 3574436305225685050 [ 8133.937293] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8133.947632] LustreError: 249644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8140.416944] Lustre: server umount lustre-MDT0001 complete [ 8152.984522] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 8156.631384] Key type lgssc unregistered [ 8156.999933] LNet: 252792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8157.011432] LNetError: 252792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8157.031967] LNet: Removed LNI 192.168.204.121@tcp [ 8158.137818] Key type .llcrypt unregistered [ 8158.139864] Key type ._llcrypt unregistered [ 8197.306551] Key type ._llcrypt registered [ 8197.310430] Key type .llcrypt registered [ 8197.561903] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8199.073488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8199.150651] alg: No test for adler32 (adler32-zlib) [ 8200.265697] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8200.527222] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 8202.256199] Key type lgssc registered [ 8203.958185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8213.281747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8213.316474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8214.679711] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8214.720705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8214.956200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8219.730842] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8229.110189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8229.296557] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8229.326868] Lustre: 254997:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8229.612787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8234.728582] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8242.364486] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8248.771703] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8255.618048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8255.807926] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8255.987575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8261.299252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8262.846208] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8270.550421] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8323.046326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8323.047038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8323.061434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8323.075402] Lustre: Skipped 1 previous similar message [ 8328.166833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8328.169754] Lustre: Skipped 1 previous similar message [ 8329.058041] Lustre: server umount lustre-OST0000 complete [ 8333.482929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8346.178569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8346.558143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8352.586328] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8352.689700] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8352.701859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8352.702994] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8352.724722] Lustre: Skipped 1 previous similar message [ 8362.046888] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8403.950211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8403.961514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8403.969189] Lustre: Skipped 1 previous similar message [ 8403.988105] Lustre: Skipped 2 previous similar messages [ 8408.401873] Lustre: server umount lustre-OST0000 complete [ 8414.178404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8414.190187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8414.191736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8414.209512] Lustre: Skipped 1 previous similar message [ 8418.326827] Lustre: server umount lustre-MDT0000 complete [ 8422.129621] LustreError: 254987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446449 with bad export cookie 7646845520142460076 [ 8422.147029] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8422.155226] LustreError: 254987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8422.524356] Lustre: server umount lustre-MDT0001 complete [ 8433.172991] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 8436.234683] Key type lgssc unregistered [ 8436.552602] LNet: 260421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8436.562811] LNetError: 260421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8437.606892] LNet: Removed LNI 192.168.204.121@tcp [ 8438.485813] Key type .llcrypt unregistered [ 8438.487843] Key type ._llcrypt unregistered [ 8464.455567] Key type ._llcrypt registered [ 8464.458263] Key type .llcrypt registered [ 8464.576977] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8465.459310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8465.535826] alg: No test for adler32 (adler32-zlib) [ 8466.578369] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8466.795958] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 8468.488346] Key type lgssc registered [ 8469.817443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8478.168965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8478.189680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8479.684026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8484.508445] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8489.957957] LustreError: 261832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8489.977371] LustreError: 261832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8493.129292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8493.628304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8497.668291] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8506.075849] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8513.176805] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8520.660080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8521.131510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8525.239115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8527.527761] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8534.838312] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8564.198957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8564.201154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8564.201769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8564.227629] Lustre: Skipped 1 previous similar message [ 8569.317565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8569.339317] Lustre: Skipped 2 previous similar messages [ 8569.966788] Lustre: server umount lustre-OST0000 complete [ 8574.437037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8574.467154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8574.475846] Lustre: Skipped 1 previous similar message [ 8579.557192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8579.571907] Lustre: Skipped 1 previous similar message [ 8584.674857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8584.683110] Lustre: Skipped 1 previous similar message [ 8588.256702] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8588.473260] Lustre: server umount lustre-MDT0000 complete [ 8589.794269] LustreError: 261833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8589.804169] LustreError: 261833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8591.871333] LustreError: 261813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446619 with bad export cookie 15438786143788237192 [ 8591.877858] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8591.884429] LustreError: 261813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8592.238855] Lustre: server umount lustre-MDT0001 complete [ 8602.005052] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 8604.664128] Key type lgssc unregistered [ 8604.972360] LNet: 266140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8604.977680] LNetError: 266140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8605.006827] LNet: Removed LNI 192.168.204.121@tcp [ 8605.691599] Key type .llcrypt unregistered [ 8605.693376] Key type ._llcrypt unregistered [ 8622.150901] Key type ._llcrypt registered [ 8622.153436] Key type .llcrypt registered [ 8622.296843] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8623.843213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8623.893475] alg: No test for adler32 (adler32-zlib) [ 8624.999132] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8625.264139] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 8626.952187] Key type lgssc registered [ 8627.874109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8635.074642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8635.090055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8636.526426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8641.286577] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8647.142809] LustreError: 267359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8647.155985] LustreError: 267359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8650.033121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8650.510323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8655.348559] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8663.793354] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8670.255601] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8677.374775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8677.739915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8681.868607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8684.317579] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8692.899857] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8707.554606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8707.561248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8707.577342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8711.655294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8711.661403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8711.664451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8713.387595] Lustre: server umount lustre-OST0000 complete [ 8717.290186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8717.299655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8717.321879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8717.329036] Lustre: Skipped 1 previous similar message [ 8721.891985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8721.893946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8721.918968] Lustre: Skipped 1 previous similar message [ 8723.617119] Lustre: server umount lustre-MDT0000 complete [ 8727.011748] LustreError: 267358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8727.028513] LustreError: 267358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8727.815087] LustreError: 267338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446755 with bad export cookie 9182444562719347996 [ 8727.820161] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8727.829137] LustreError: 267338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8728.154760] Lustre: server umount lustre-MDT0001 complete [ 8738.014294] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 8740.741212] Key type lgssc unregistered [ 8741.087055] LNet: 271039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8741.102483] LNetError: 271039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8741.124827] LNet: Removed LNI 192.168.204.121@tcp [ 8741.901334] Key type .llcrypt unregistered [ 8741.904607] Key type ._llcrypt unregistered [ 8764.364956] Key type ._llcrypt registered [ 8764.371523] Key type .llcrypt registered [ 8764.496154] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8765.643926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8765.697123] alg: No test for adler32 (adler32-zlib) [ 8766.864267] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8767.131639] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 8768.817138] Key type lgssc registered [ 8769.835771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8778.199298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8778.234188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8779.831562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8784.919989] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8789.987733] LustreError: 272449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8790.012660] LustreError: 272449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8793.431039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8793.712103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8798.258846] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8806.074733] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8812.360810] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8819.008403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8819.361224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8824.524396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8825.882549] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8834.057099] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8856.549819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8856.561495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8856.580628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8860.654308] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8860.657091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8860.662898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8862.396299] Lustre: server umount lustre-OST0000 complete [ 8870.369792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8870.373807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8870.385228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8870.394452] Lustre: Skipped 1 previous similar message [ 8872.249758] Lustre: server umount lustre-MDT0000 complete [ 8875.358584] LustreError: 272429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446903 with bad export cookie 14764644597971281890 [ 8875.367946] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8875.374024] LustreError: 272429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8875.688408] Lustre: server umount lustre-MDT0001 complete [ 8885.294664] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 8887.714956] Key type lgssc unregistered [ 8888.038654] LNet: 276752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8888.044686] LNetError: 276752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8888.054574] LNet: Removed LNI 192.168.204.121@tcp [ 8888.849398] Key type .llcrypt unregistered [ 8888.852173] Key type ._llcrypt unregistered [ 8903.888588] Key type ._llcrypt registered [ 8903.890446] Key type .llcrypt registered [ 8903.989153] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8904.822195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8904.972580] alg: No test for adler32 (adler32-zlib) [ 8905.983373] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8906.203260] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 8907.888609] Key type lgssc registered [ 8908.865398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8916.566677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8916.611386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8918.155809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8922.570426] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8927.720747] LustreError: 277962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8927.744524] LustreError: 277962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8932.119303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8932.627338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8937.652986] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8946.649626] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8951.879972] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8957.944030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8958.280809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8962.422251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8964.131698] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8970.644612] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8993.251081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8993.254286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8993.256462] LustreError: Skipped 1 previous similar message [ 8993.262368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8993.289229] Lustre: Skipped 1 previous similar message [ 8993.305125] Lustre: Skipped 1 previous similar message [ 8996.605514] Lustre: server umount lustre-OST0000 complete [ 9003.491706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9003.504228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9003.509508] Lustre: Skipped 1 previous similar message [ 9008.612321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9008.618219] Lustre: Skipped 2 previous similar messages [ 9013.747181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9013.758553] Lustre: Skipped 1 previous similar message [ 9014.240330] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9014.392647] Lustre: server umount lustre-MDT0000 complete [ 9017.695882] LustreError: 279830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447045 with bad export cookie 664397185413655989 [ 9017.701307] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9017.706044] LustreError: 279830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9018.019353] Lustre: server umount lustre-MDT0001 complete [ 9027.210169] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 9029.918301] Key type lgssc unregistered [ 9030.241159] LNet: 281685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9030.248363] LNetError: 281685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9030.264898] LNet: Removed LNI 192.168.204.121@tcp [ 9031.082516] Key type .llcrypt unregistered [ 9031.085900] Key type ._llcrypt unregistered [ 9041.392530] Key type ._llcrypt registered [ 9041.395144] Key type .llcrypt registered [ 9041.489950] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9057.651632] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 9072.000052] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 9073.330562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9073.417624] alg: No test for adler32 (adler32-zlib) [ 9074.562501] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9074.905330] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 9076.641072] Key type lgssc registered [ 9078.254635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9114.438755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9114.462077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9115.736756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9115.770786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9115.869314] Lustre: lustre-MDT0000: new disk, initializing [ 9116.010120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9116.021259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9121.160810] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9131.676275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9131.830971] Lustre: 285249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9131.887576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9131.890605] Lustre: Skipped 1 previous similar message [ 9131.942328] Lustre: lustre-MDT0001: new disk, initializing [ 9132.021045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9132.048742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9132.060840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9136.870792] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9147.368632] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9153.865776] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9161.433426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9161.770539] Lustre: lustre-OST0000: new disk, initializing [ 9161.774642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9161.781989] Lustre: 287218:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9161.853760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9167.928937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9167.936101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9168.140137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9168.732517] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9178.954491] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9183.770536] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9184.020084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9189.291694] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9189.460852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9193.961556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9193.963242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9193.968305] LustreError: Skipped 1 previous similar message [ 9193.971296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9193.978421] Lustre: Skipped 1 previous similar message [ 9194.009900] Lustre: Skipped 1 previous similar message [ 9197.908617] Lustre: server umount lustre-OST0000 complete [ 9203.682197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9203.689172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9203.709070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9208.461711] Lustre: server umount lustre-MDT0000 complete [ 9212.035237] LustreError: 285241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447239 with bad export cookie 12612386069397589728 [ 9212.042316] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9212.043892] LustreError: 285241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9212.438734] Lustre: server umount lustre-MDT0001 complete [ 9220.144697] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:54:07 (1788447247) [ 9225.874152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9226.251787] LustreError: 289630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9226.274754] LustreError: 289630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9226.337379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9230.803473] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9231.343695] LustreError: 289631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9235.434107] LustreError: 289630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9239.429066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9239.787745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9244.507278] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9251.542878] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9256.745923] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9263.235474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9263.721369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9269.617698] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9276.789076] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9285.606146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9285.615206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9285.634928] Lustre: Skipped 1 previous similar message [ 9285.641085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9285.646308] Lustre: Skipped 2 previous similar messages [ 9291.631667] Lustre: server umount lustre-OST0000 complete [ 9296.353250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9296.358373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9296.374802] Lustre: Skipped 1 previous similar message [ 9296.381397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9296.391855] Lustre: Skipped 2 previous similar messages [ 9301.986365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9301.992043] Lustre: Skipped 2 previous similar messages [ 9309.152360] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9311.339588] Lustre: server umount lustre-MDT0000 complete [ 9312.226132] LustreError: 289630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9312.246079] LustreError: 289630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9314.437034] LustreError: 289610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447342 with bad export cookie 12612386069397590610 [ 9314.444881] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9314.447927] LustreError: 289610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9314.745735] Lustre: server umount lustre-MDT0001 complete [ 9323.997421] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 9326.995279] Key type lgssc unregistered [ 9327.306758] LNet: 293167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9327.320875] LNetError: 293167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9327.340254] LNet: Removed LNI 192.168.204.121@tcp [ 9328.151250] Key type .llcrypt unregistered [ 9328.153696] Key type ._llcrypt unregistered [ 9352.183794] Key type ._llcrypt registered [ 9352.186430] Key type .llcrypt registered [ 9352.274278] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 9366.686418] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 9367.849926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9367.884313] alg: No test for adler32 (adler32-zlib) [ 9369.012425] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9369.251494] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 9370.944243] Key type lgssc registered [ 9372.013691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9419.010103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9419.054995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9420.349615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9420.392578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9420.493242] Lustre: lustre-MDT0000: new disk, initializing [ 9420.634774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9420.656513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9425.687820] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9439.700198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9439.878704] Lustre: 296989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9440.020299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9440.024352] Lustre: Skipped 1 previous similar message [ 9440.125821] Lustre: lustre-MDT0001: new disk, initializing [ 9440.231386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9440.264706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9440.273389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9444.413231] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9455.039395] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9461.140135] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9468.355321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9468.636891] Lustre: lustre-OST2710: new disk, initializing [ 9468.650849] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9468.657986] Lustre: 298956:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9468.828536] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9474.126786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9474.155736] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9474.358719] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9475.853293] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9486.480634] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9495.112709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9495.321967] Lustre: lustre-OST03e8: new disk, initializing [ 9495.327344] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9495.333216] Lustre: 300025:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9495.495423] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9501.883427] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9503.870104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9503.893244] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9503.998969] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9509.759754] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9516.080614] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9516.285138] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9520.133821] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9520.330445] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9523.542589] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9523.707588] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9529.258891] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9529.508606] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9534.959608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9534.973968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9535.002723] Lustre: Skipped 2 previous similar messages [ 9535.016177] Lustre: Skipped 1 previous similar message [ 9536.544571] Lustre: server umount lustre-MDT0000 complete [ 9540.928209] LustreError: 296980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447568 with bad export cookie 91105870105634500 [ 9540.935248] LustreError: 296980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9540.935952] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9541.388547] Lustre: server umount lustre-MDT0001 complete [ 9556.244540] Lustre: server umount lustre-OST2710 complete [ 9570.131575] Lustre: server umount lustre-OST03e8 complete [ 9589.113647] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 9599.625177] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 9639.325288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9639.646614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9639.695645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9639.753707] Lustre: lustre-MDT0000: new disk, initializing [ 9639.856138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9639.885562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9645.142938] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9656.777848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9656.913794] Lustre: 305311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9657.019715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9657.020955] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9657.024915] Lustre: Skipped 1 previous similar message [ 9657.034060] Lustre: Skipped 2 previous similar messages [ 9657.127950] Lustre: lustre-MDT0001: new disk, initializing [ 9657.195594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9657.230821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9657.237219] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9662.865686] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9673.052392] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9678.079711] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9684.081218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9684.369504] Lustre: lustre-OST0000: new disk, initializing [ 9684.373845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9684.385029] Lustre: 307255:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9684.456572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9689.744855] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9692.829075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9692.836279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9692.927211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9698.986349] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9704.207669] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9704.473585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9710.521820] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9710.931126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9718.758381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9718.761283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9718.774335] LustreError: Skipped 1 previous similar message [ 9718.790135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9718.792935] Lustre: Skipped 2 previous similar messages [ 9718.806866] Lustre: Skipped 1 previous similar message [ 9720.181443] Lustre: server umount lustre-OST0000 complete [ 9728.993768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9728.996446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9729.008518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9729.016661] Lustre: Skipped 1 previous similar message [ 9730.892297] Lustre: server umount lustre-MDT0000 complete [ 9734.118854] LustreError: 308762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9734.125643] LustreError: 308762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9734.374826] LustreError: 307254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447762 with bad export cookie 91105870105636180 [ 9734.383405] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9734.388172] LustreError: 307254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9734.690165] Lustre: server umount lustre-MDT0001 complete [ 9744.434395] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:02:51 (1788447771) [ 9746.048606] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9747.985287] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:02:54 (1788447774) [ 9751.112902] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 9778.690828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9779.281455] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9779.315327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9779.595940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9784.800540] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9794.162519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9794.282189] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9794.324726] Lustre: 311113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9799.364629] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9822.190984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9822.360474] Lustre: Denying initial registration attempt from nid 192.168.204.121@tcp, specified as failover [ 9822.375932] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9822.390073] LustreError: 312567:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9822.394689] LustreError: 312567:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9822.399428] LustreError: 312567:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9822.466944] Lustre: server umount lustre-OST0000 complete [ 9822.469821] LustreError: 312567:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9830.379805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9830.387508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9830.395363] Lustre: Skipped 1 previous similar message [ 9830.402708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9834.911619] Lustre: server umount lustre-MDT0000 complete [ 9837.537396] LustreError: 312233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9837.570465] LustreError: 312233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9838.451089] LustreError: 311106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447866 with bad export cookie 91105870105637069 [ 9838.462713] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9838.470712] LustreError: 311106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9838.738201] Lustre: server umount lustre-MDT0001 complete [ 9848.349107] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 9850.364399] Key type lgssc unregistered [ 9850.634847] LNet: 313689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9850.640406] LNetError: 313689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9851.692378] LNet: Removed LNI 192.168.204.121@tcp [ 9852.489505] Key type .llcrypt unregistered [ 9852.499624] Key type ._llcrypt unregistered [ 9863.625858] Key type ._llcrypt registered [ 9863.628185] Key type .llcrypt registered [ 9863.716864] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 9864.520825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9864.705959] alg: No test for adler32 (adler32-zlib) [ 9865.701708] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9865.904529] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 9867.584503] Key type lgssc registered [ 9868.528758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9918.213230] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 9928.461747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9928.488176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9929.732883] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9929.755215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9929.983553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9935.500051] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9943.860327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9943.958228] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9944.023576] Lustre: 316447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9944.337978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9948.508037] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9957.942330] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9965.198781] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9986.288492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9986.527967] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9986.750954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9992.452823] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10000.233604] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10004.963000] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10004.976538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10004.990559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10004.998655] Lustre: Skipped 1 previous similar message [10008.806833] Lustre: server umount lustre-OST0000 complete [10015.209295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10015.211299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10015.226230] Lustre: Skipped 2 previous similar messages [10018.960310] Lustre: server umount lustre-MDT0000 complete [10022.629354] LustreError: 317190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448050 with bad export cookie 12511058896024718074 [10022.630294] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10022.642251] LustreError: 317190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10022.891394] Lustre: server umount lustre-MDT0001 complete [10031.789573] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [10034.334053] Key type lgssc unregistered [10034.673891] LNet: 320022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10034.687928] LNetError: 320022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10034.715895] LNet: Removed LNI 192.168.204.121@tcp [10035.380954] Key type .llcrypt unregistered [10035.383141] Key type ._llcrypt unregistered [10058.481126] Key type ._llcrypt registered [10058.486967] Key type .llcrypt registered [10058.567855] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [10059.339816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10059.426480] alg: No test for adler32 (adler32-zlib) [10060.392147] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10060.618394] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [10062.272191] Key type lgssc registered [10063.207945] Lustre: Echo OBD driver; http://www.lustre.org/ [10070.649419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10070.665631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10072.108258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10076.160892] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10082.278132] LustreError: 321435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10082.299961] LustreError: 321435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10086.074352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10086.494915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10091.056056] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10100.263257] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10106.036608] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10114.543831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10114.984812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10121.136506] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10129.088534] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10151.794347] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.21@tcp (stopping) [10151.920655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10151.949626] Lustre: Skipped 2 previous similar messages [10154.939880] Lustre: server umount lustre-MDT0000 complete [10156.779734] LustreError: 321429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10160.824991] LustreError: 321416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448188 with bad export cookie 5221282890801163895 [10160.830355] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10160.842967] LustreError: 321416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10161.886494] LustreError: 323394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10161.908220] LustreError: 323394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10161.910632] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.21@tcp (stopping) [10161.926852] Lustre: Skipped 3 previous similar messages [10162.177890] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10165.218084] LustreError: 321429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10165.220945] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10165.263491] LustreError: 321429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10165.294381] Lustre: Skipped 1 previous similar message [10167.210326] Lustre: server umount lustre-MDT0001 complete [10173.824865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10183.229077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10185.504177] LustreError: 325033:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10185.515929] LustreError: 325033:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@0000000057125968 x1875323742369536/t0(0) o250->MGC192.168.204.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788448213 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10185.539935] LustreError: 325033:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [10185.696755] LustreError: 320743:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000051929d6b x1875323742369920/t0(0) o250->MGC192.168.204.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10185.987840] LustreError: 323266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10186.006530] LustreError: 323266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10186.109658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10186.172793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10190.713781] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10196.328846] LustreError: 325047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10196.338254] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10196.364690] LustreError: 325047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10200.270636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10200.802941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10205.331536] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10206.212921] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10215.305250] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10216.939868] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10223.814968] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10225.893603] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10231.779198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10231.789185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10231.806976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10231.816223] Lustre: Skipped 2 previous similar messages [10236.914949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10236.923862] Lustre: Skipped 1 previous similar message [10237.503415] Lustre: server umount lustre-OST0000 complete [10242.018198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10242.026833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10242.038862] Lustre: Skipped 2 previous similar messages [10247.140578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10247.150595] Lustre: Skipped 3 previous similar messages [10249.489275] Lustre: server umount lustre-MDT0000 complete [10252.259751] LustreError: 325051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10252.288655] LustreError: 325051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10253.587103] LustreError: 321415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448281 with bad export cookie 5221282890801171679 [10253.590229] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10253.603152] LustreError: 321415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10253.922747] Lustre: server umount lustre-MDT0001 complete [10266.001898] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [10269.563953] Key type lgssc unregistered [10269.860235] LNet: 327720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10269.874518] LNetError: 327720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10269.896053] LNet: Removed LNI 192.168.204.121@tcp [10270.935926] Key type .llcrypt unregistered [10270.941893] Key type ._llcrypt unregistered [10296.466605] Key type ._llcrypt registered [10296.468282] Key type .llcrypt registered [10296.559328] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [10297.607962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10297.630670] alg: No test for adler32 (adler32-zlib) [10298.639799] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10298.845222] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [10300.554425] Key type lgssc registered [10301.736387] Lustre: Echo OBD driver; http://www.lustre.org/ [10310.202494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10310.248135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10311.724951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10316.565522] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10321.896517] LustreError: 329136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10321.931074] LustreError: 329136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10326.611311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10327.003351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10331.987246] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10339.859204] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10345.873668] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10348.488677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10348.497419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10348.500547] Lustre: Skipped 1 previous similar message [10348.516660] Lustre: Skipped 1 previous similar message [10351.588537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10351.599585] Lustre: Skipped 1 previous similar message [10354.863244] Lustre: server umount lustre-MDT0000 complete [10356.711770] LustreError: 329133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10356.733154] LustreError: 329133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10358.976122] LustreError: 329117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448386 with bad export cookie 8935524198805202577 [10358.987096] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10358.988119] LustreError: 329117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10359.347831] Lustre: server umount lustre-MDT0001 complete [10375.955720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10376.297740] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10376.336392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10376.616333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10382.659471] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10392.930192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10393.095408] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10393.132781] Lustre: 331726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10393.412936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10398.478419] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10405.633647] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10411.793524] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10422.588075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10422.793588] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10422.800909] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10422.812719] LustreError: 333610:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10422.818879] LustreError: 333610:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10422.825389] LustreError: 333610:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10422.900424] Lustre: server umount lustre-OST0000 complete [10422.905364] LustreError: 333610:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10434.161026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10434.426624] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10434.597379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10439.849196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10440.946798] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10449.553168] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10457.704748] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10457.924710] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10458.137969] Lustre: lustre-OST0001: new disk, initializing [10458.145835] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10458.165355] Lustre: 335153:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10458.277498] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10463.852033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10463.874513] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10463.945667] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10463.998459] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10472.325946] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10479.590421] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10479.591651] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10479.611693] LustreError: Skipped 1 previous similar message [10479.633798] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10481.693122] Lustre: server umount lustre-OST0001 complete [10484.714625] LustreError: 334077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10484.726659] LustreError: 334077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10489.836287] LustreError: 334388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10489.840277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10489.843652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10489.843659] Lustre: Skipped 1 previous similar message [10489.856556] LustreError: 334388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10489.941286] Lustre: Skipped 2 previous similar messages [10492.948845] Lustre: server umount lustre-OST0000 complete [10500.068169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10500.071982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10500.081565] Lustre: Skipped 1 previous similar message [10500.092394] Lustre: Skipped 2 previous similar messages [10503.813378] Lustre: server umount lustre-MDT0000 complete [10505.185202] LustreError: 332815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10505.203869] LustreError: 332815:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10508.157818] LustreError: 334092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448535 with bad export cookie 8935524198805203130 [10508.172672] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10508.173267] LustreError: 334092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10508.455292] Lustre: server umount lustre-MDT0001 complete [10522.002988] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [10525.897598] Key type lgssc unregistered [10526.214659] LNet: 337109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10526.221346] LNetError: 337109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10526.237843] LNet: Removed LNI 192.168.204.121@tcp [10526.976297] Key type .llcrypt unregistered [10526.983157] Key type ._llcrypt unregistered [10610.840712] Key type ._llcrypt registered [10610.842665] Key type .llcrypt registered [10610.984979] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [10628.788353] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [10630.267269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10630.309410] alg: No test for adler32 (adler32-zlib) [10631.512328] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10631.977172] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [10633.704178] Key type lgssc registered [10634.917774] Lustre: Echo OBD driver; http://www.lustre.org/ [10674.947389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10674.965528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10676.229130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10676.266065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10676.357581] Lustre: lustre-MDT0000: new disk, initializing [10676.435049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10676.446417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10681.348244] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10691.252489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10691.384446] Lustre: 342048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10691.459093] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10691.464454] Lustre: Skipped 1 previous similar message [10691.586747] Lustre: lustre-MDT0001: new disk, initializing [10691.714138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10691.748549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10691.770830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10696.459549] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10706.311825] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10713.304796] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10720.864955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10721.180585] Lustre: lustre-OST0000: new disk, initializing [10721.190729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10721.195534] Lustre: 344019:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10721.332715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10727.470785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10727.482691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10727.613865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10727.692268] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10738.705259] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10744.142513] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10744.441319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10749.350300] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10749.567914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10753.505781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10753.519926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10753.522795] LustreError: Skipped 1 previous similar message [10753.532012] Lustre: Skipped 1 previous similar message [10753.534299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10753.540742] Lustre: Skipped 1 previous similar message [10757.737515] Lustre: server umount lustre-OST0000 complete [10763.747040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10763.750030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10763.760820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10763.777892] Lustre: Skipped 1 previous similar message [10767.803940] Lustre: server umount lustre-MDT0000 complete [10771.708884] LustreError: 342040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448799 with bad export cookie 1225158625147375958 [10771.723422] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10771.726519] LustreError: 342040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10772.049450] Lustre: server umount lustre-MDT0001 complete [10782.721838] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:20:09 (1788448809) [10792.356160] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:20:18 (1788448818) [10800.745384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10801.201075] LustreError: 346774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10801.223556] LustreError: 346774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10801.282424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10805.157234] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10806.755127] LustreError: 346775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10810.850699] LustreError: 346774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10813.760338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10814.022403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10818.075782] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10825.979547] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10832.096614] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10840.173459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10840.652220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10848.033409] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10857.077565] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10866.433932] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10869.223551] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10871.249076] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10873.757214] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10876.088035] Lustre: Failing over lustre-MDT0000 [10876.763504] Lustre: server umount lustre-MDT0000 complete [10879.972338] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10879.973886] LustreError: 346771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10880.007126] Lustre: Skipped 2 previous similar messages [10880.027245] LustreError: 346771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10885.094336] LustreError: 346775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10885.136742] LustreError: 346775:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10895.329840] LustreError: 347502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10895.331328] Lustre: 340270:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788448907/real 1788448907] req@000000008b3558cb x1875324341133696/t0(0) o400->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788448923 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10895.375604] LustreError: 347502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10895.456803] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10898.843382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10906.090408] LustreError: 340268:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b3b03d5e x1875324341139200/t0(0) o250->MGC192.168.204.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10906.400395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10906.454448] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10906.859061] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10910.630603] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10911.732283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10911.774190] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10911.812802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10921.427853] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10923.359774] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10927.714068] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10930.307830] Lustre: Failing over lustre-MDT0000 [10930.720041] Lustre: server umount lustre-MDT0000 complete [10931.462485] LustreError: 349223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10931.492529] LustreError: 349223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10932.196522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10932.216424] Lustre: Skipped 2 previous similar messages [10947.488166] Lustre: 340271:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788448959/real 1788448959] req@000000001ef406de x1875324341154304/t0(0) o400->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788448975 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10947.520885] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10950.010129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10957.803381] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1100a347456a5bf2 [10957.835521] Lustre: MGC192.168.204.121@tcp: Connection restored to 0@lo (at 0@lo) [10957.847509] Lustre: Skipped 2 previous similar messages [10958.207305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10958.279388] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10962.148763] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10962.960813] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10963.434398] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10963.459508] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10963.497575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10974.922485] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10976.819387] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10983.910882] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10983.918249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10983.918385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10983.918390] Lustre: Skipped 1 previous similar message [10983.933881] Lustre: Skipped 1 previous similar message [10988.736932] Lustre: server umount lustre-MDT0000 complete [10989.033443] LustreError: 347502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10989.057317] LustreError: 347502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [10992.621101] LustreError: 347509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449020 with bad export cookie 1225158625147378674 [10992.633580] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10992.639954] LustreError: 347509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10993.073994] Lustre: server umount lustre-MDT0001 complete [11007.187187] Lustre: server umount lustre-OST0000 complete [11024.973664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11025.488776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11029.831895] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11039.257577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11044.641390] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11052.940556] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11059.476847] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11067.340946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11070.897757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11075.662743] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11085.819568] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11095.527971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11095.545600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11095.547773] Lustre: Skipped 1 previous similar message [11095.550351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11095.550358] Lustre: Skipped 2 previous similar messages [11098.729386] Lustre: server umount lustre-OST0000 complete [11105.765883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11105.772059] Lustre: Skipped 2 previous similar messages [11109.046728] Lustre: server umount lustre-MDT0000 complete [11110.884977] LustreError: 353850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11110.911617] LustreError: 353850:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11113.161659] LustreError: 353102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449140 with bad export cookie 1225158625147379423 [11113.177754] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11113.509800] Lustre: server umount lustre-MDT0001 complete [11125.498518] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [11128.359494] Key type lgssc unregistered [11128.707980] LNet: 356655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11128.719188] LNetError: 356655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11128.743874] LNet: Removed LNI 192.168.204.121@tcp [11129.713411] Key type .llcrypt unregistered [11129.716306] Key type ._llcrypt unregistered [11158.376361] Key type ._llcrypt registered [11158.380227] Key type .llcrypt registered [11158.543520] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [11160.117640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11160.155933] alg: No test for adler32 (adler32-zlib) [11161.320236] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11161.684480] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [11163.408217] Key type lgssc registered [11164.795230] Lustre: Echo OBD driver; http://www.lustre.org/ [11173.265875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11173.302831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11174.751127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11180.420883] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11183.013173] LustreError: 358183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11183.026676] LustreError: 358183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11185.074180] LustreError: 358184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11189.878550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11190.349091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11195.474397] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11202.949732] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11209.453975] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11216.509826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11217.027081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11220.098046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11223.251894] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11231.727884] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11245.027070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11245.036356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11245.058632] Lustre: Skipped 2 previous similar messages [11245.083976] Lustre: Skipped 2 previous similar messages [11246.041943] Lustre: server umount lustre-MDT0000 complete [11250.152273] LustreError: 358184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11250.200496] LustreError: 358184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11250.818667] LustreError: 358163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449278 with bad export cookie 11054049566417073742 [11250.824614] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11250.843340] LustreError: 358163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11251.085393] Lustre: server umount lustre-MDT0001 complete [11267.128325] Lustre: server umount lustre-OST0000 complete [11282.485880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11283.040016] LustreError: 361897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11283.114473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11288.089557] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11288.545954] LustreError: 361898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11297.768253] LustreError: 361897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11297.801026] LustreError: 361897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11298.966534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11299.489867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11304.554986] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11313.750190] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11320.074422] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11327.771053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11328.046960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11335.110649] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11344.404388] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11352.629767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11370.979165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11370.983731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11370.992837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11373.032771] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11373.033826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11373.053884] Lustre: Skipped 2 previous similar messages [11374.978629] Lustre: server umount lustre-MDT0000 complete [11378.158633] LustreError: 361898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11378.195024] LustreError: 361898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11379.164392] LustreError: 361877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449406 with bad export cookie 11054049566417074855 [11379.165698] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11379.176145] LustreError: 361877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11379.602663] Lustre: server umount lustre-MDT0001 complete [11394.127021] Lustre: server umount lustre-OST0000 complete [11409.167138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11409.641782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11414.596953] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11415.011177] LustreError: 365834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11415.024180] LustreError: 365834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11424.543795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11428.976195] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11437.283695] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11444.922523] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11452.220253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11452.656282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11452.661245] Lustre: Skipped 1 previous similar message [11454.730035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11460.005529] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11467.877138] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11475.434335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11475.437209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11475.443826] LustreError: Skipped 1 previous similar message [11475.444653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11475.492430] Lustre: Skipped 2 previous similar messages [11479.886372] Lustre: server umount lustre-OST0000 complete [11485.672131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11485.675758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11485.695029] Lustre: Skipped 1 previous similar message [11485.711193] Lustre: Skipped 2 previous similar messages [11490.785478] LustreError: 365828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11490.814117] LustreError: 365828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11492.177193] Lustre: server umount lustre-MDT0000 complete [11496.798606] LustreError: 365814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449524 with bad export cookie 11054049566417075877 [11496.803997] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11496.810533] LustreError: 365814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11497.204715] Lustre: server umount lustre-MDT0001 complete [11507.737705] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [11510.317075] Key type lgssc unregistered [11510.618550] LNet: 369365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11510.627472] LNetError: 369365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11510.641972] LNet: Removed LNI 192.168.204.121@tcp [11511.588155] Key type .llcrypt unregistered [11511.590282] Key type ._llcrypt unregistered [11539.984338] Key type ._llcrypt registered [11539.986764] Key type .llcrypt registered [11540.085880] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [11540.720811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11540.806441] alg: No test for adler32 (adler32-zlib) [11542.029448] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11542.322966] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [11544.088178] Key type lgssc registered [11545.629886] Lustre: Echo OBD driver; http://www.lustre.org/ [11553.665886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11553.681980] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11553.686549] LustreError: 370802:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11553.704428] LustreError: 370802:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11553.711146] LustreError: 370802:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11553.719957] LustreError: 370802:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11553.732230] LustreError: 370802:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11561.307991] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11561.328324] LustreError: 371199:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11561.383478] LustreError: 371199:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11561.409169] LustreError: 371199:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11561.436725] LustreError: 371199:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11561.442662] LustreError: 371199:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11580.248528] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [11583.325455] Key type lgssc unregistered [11583.700758] LNet: 372202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11583.713817] LNetError: 372202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11583.732262] LNet: Removed LNI 192.168.204.121@tcp [11584.559611] Key type .llcrypt unregistered [11584.565478] Key type ._llcrypt unregistered [11605.510727] Key type ._llcrypt registered [11605.512984] Key type .llcrypt registered [11605.649832] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [11619.656678] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [11620.847813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11620.973310] alg: No test for adler32 (adler32-zlib) [11622.128473] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11622.568897] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [11624.352935] Key type lgssc registered [11625.731508] Lustre: Echo OBD driver; http://www.lustre.org/ [11674.664868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11674.730124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11676.012253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11676.097258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11676.307389] Lustre: lustre-MDT0000: new disk, initializing [11676.467995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11676.509079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11681.396671] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11691.995652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11692.123221] Lustre: 375374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11692.206724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11692.213552] Lustre: Skipped 1 previous similar message [11692.322966] Lustre: lustre-MDT0001: new disk, initializing [11692.462060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11692.557901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11692.577850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11697.095905] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11707.072839] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11712.900801] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11718.905143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11719.234699] Lustre: lustre-OST0000: new disk, initializing [11719.240550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11719.249944] Lustre: 377343:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11719.374907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11725.376942] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11726.882697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11726.890624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11727.056129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11734.221362] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11739.509619] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11739.758739] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11745.135229] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11745.456678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11752.930521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11752.931925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11752.941787] LustreError: Skipped 1 previous similar message [11752.942877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11752.964469] Lustre: Skipped 1 previous similar message [11758.054398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11758.068470] Lustre: Skipped 2 previous similar messages [11762.144259] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11762.333575] Lustre: server umount lustre-OST0000 complete [11768.293624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11768.313696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11768.324898] Lustre: Skipped 1 previous similar message [11772.937822] Lustre: server umount lustre-MDT0000 complete [11777.896903] LustreError: 375367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449805 with bad export cookie 17766088734708831182 [11777.899544] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11777.906218] LustreError: 375367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11778.445183] Lustre: server umount lustre-MDT0001 complete [11789.346862] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11791.423445] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:36:57 (1788449817) [11799.648981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11800.110225] LustreError: 379805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11800.127068] LustreError: 379805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11800.186823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11804.322657] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11805.674714] LustreError: 379806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11809.766610] LustreError: 379805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11812.446236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11812.865414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11817.007686] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11823.900407] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11828.910200] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11835.293268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11835.669641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11841.878534] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11848.728273] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11855.719244] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11855.934858] Lustre: lustre-OST0001: new disk, initializing [11855.939569] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11855.953839] Lustre: 382641:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11856.071188] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11862.910705] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11866.236627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11866.248311] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11866.365184] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11872.454555] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11876.834080] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11876.841211] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11876.841287] LustreError: Skipped 1 previous similar message [11876.849072] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11876.861567] Lustre: Skipped 2 previous similar messages [11881.961551] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11881.970694] Lustre: Skipped 3 previous similar messages [11882.226554] Lustre: server umount lustre-OST0001 complete [11884.289162] LustreError: 381626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11892.202042] LustreError: 381626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11892.222243] LustreError: 381626:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11899.361364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11899.367769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11899.377288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11901.412123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11902.444180] LustreError: 381626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11902.463945] LustreError: 381626:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11904.752425] Lustre: server umount lustre-OST0000 complete [11910.118125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11910.127762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11910.142287] Lustre: Skipped 1 previous similar message [11915.241321] Lustre: server umount lustre-MDT0000 complete [11919.258309] LustreError: 382640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449946 with bad export cookie 17766088734708832064 [11919.262444] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11919.271357] LustreError: 382640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11925.748350] Lustre: server umount lustre-MDT0001 complete [11938.065813] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [11940.818693] Key type lgssc unregistered [11941.109346] LNet: 384629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11941.124694] LNetError: 384629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11941.153981] LNet: Removed LNI 192.168.204.121@tcp [11942.107192] Key type .llcrypt unregistered [11942.109158] Key type ._llcrypt unregistered [11991.164811] Key type ._llcrypt registered [11991.167320] Key type .llcrypt registered [11991.367408] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [11992.965771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11993.041511] alg: No test for adler32 (adler32-zlib) [11994.197312] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11994.535182] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [11996.248437] Key type lgssc registered [11997.504885] Lustre: Echo OBD driver; http://www.lustre.org/ [12005.532695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12005.543398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12006.733201] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12006.754132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12006.895420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12011.838294] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12021.604902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12021.710868] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12021.734988] Lustre: 387427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12021.976213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12027.107816] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12034.863550] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12041.239865] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12046.307266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12046.312244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12046.332034] Lustre: Skipped 1 previous similar message [12046.341103] Lustre: Skipped 1 previous similar message [12049.927698] Lustre: server umount lustre-MDT0000 complete [12054.083219] LustreError: 387417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450081 with bad export cookie 18412083235128631511 [12054.085220] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12054.097280] LustreError: 387417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12054.346900] Lustre: server umount lustre-MDT0001 complete [12063.077133] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:41:29 (1788450089) [12070.323242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12071.067565] LustreError: 389935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12071.093733] LustreError: 389935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12071.151826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12076.519138] LustreError: 389935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12077.075226] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12081.641708] LustreError: 389936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12086.755236] LustreError: 389935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12087.276508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12087.646938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12093.346624] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12102.836814] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12109.901271] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12118.838846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12119.115197] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12119.419370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12127.154932] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12136.142176] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12144.868343] Lustre: Permanently deactivating lustre-OST0000 [12144.905669] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12153.940872] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12156.101782] LustreError: 392924:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12156.117943] LustreError: 392924:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12156.128327] LustreError: 392924:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12159.894725] LustreError: 392973:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12159.900620] LustreError: 392973:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12168.845379] Lustre: server umount lustre-OST0000 complete [12172.212351] LustreError: 393223:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12172.224810] LustreError: 393223:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12174.818862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12174.830280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12174.841959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12178.915267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12178.921205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12178.960686] Lustre: Skipped 1 previous similar message [12184.039610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12184.057381] Lustre: Skipped 1 previous similar message [12185.147908] Lustre: server umount lustre-MDT0000 complete [12189.161258] LustreError: 389930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12189.193770] LustreError: 389930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12189.667219] LustreError: 389916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450217 with bad export cookie 18412083235128632092 [12189.673453] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12189.678705] LustreError: 389916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12200.054229] Lustre: server umount lustre-MDT0001 complete [12210.209767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12215.535589] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12218.542125] LustreError: 394383:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12225.390497] Lustre: 394723:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.121@tcp [12227.501530] Lustre: server umount MGS complete [12238.729525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12239.208496] LustreError: 395447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12239.231122] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12239.236506] Lustre: Skipped 1 previous similar message [12239.269352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12243.606712] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12253.620511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12253.860369] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12258.352588] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12268.929000] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12276.091337] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12277.923849] Lustre: Permanently reactivating lustre-OST0000 [12277.941296] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12277.946204] Lustre: Skipped 2 previous similar messages [12304.353348] LustreError: 397182:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12311.855264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12312.293664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12312.304757] Lustre: Skipped 1 previous similar message [12318.046691] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12325.206722] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12331.242652] Lustre: server umount lustre-OST0000 complete [12336.099243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12336.108246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12336.124449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12340.194132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12345.315065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12345.321526] Lustre: Skipped 3 previous similar messages [12346.455347] Lustre: server umount lustre-MDT0000 complete [12349.410390] Lustre: 386747:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450361/real 1788450361] req@00000000a4dbea5b x1875325770371328/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788450377 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12349.431598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12349.439920] LustreError: 395447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12349.452231] LustreError: 395447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12350.434491] Lustre: 386749:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450362/real 1788450362] req@00000000334b62e4 x1875325770372096/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788450378 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12351.343867] LustreError: 395428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450379 with bad export cookie 18412083235128633142 [12351.354690] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12351.358148] LustreError: 395428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12351.892077] Lustre: server umount lustre-MDT0001 complete [12365.371676] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [12368.175899] Key type lgssc unregistered [12368.511771] LNet: 399762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12368.515063] LNetError: 399762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12368.529111] LNet: Removed LNI 192.168.204.121@tcp [12369.397148] Key type .llcrypt unregistered [12369.401731] Key type ._llcrypt unregistered [12388.943880] Key type ._llcrypt registered [12388.946201] Key type .llcrypt registered [12389.039338] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [12402.930021] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [12404.036652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12404.099458] alg: No test for adler32 (adler32-zlib) [12405.124842] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12405.385869] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [12407.056363] Key type lgssc registered [12408.500879] Lustre: Echo OBD driver; http://www.lustre.org/ [12444.484789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12444.505564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12445.646930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12445.687891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12445.834553] Lustre: lustre-MDT0000: new disk, initializing [12445.931062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12445.955746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12450.398200] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12463.795666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12463.982712] Lustre: 402916:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12464.040144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12464.049175] Lustre: Skipped 1 previous similar message [12464.224470] Lustre: lustre-MDT0001: new disk, initializing [12464.398794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12464.470908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12464.506602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12471.282050] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12481.763708] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12487.997870] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12495.014696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12495.241798] Lustre: lustre-OST0000: new disk, initializing [12495.245290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12495.255890] Lustre: 404885:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12495.345707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12500.610861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12500.628448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12500.753305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12502.076839] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12511.253839] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12516.403876] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12516.601607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12522.110647] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12522.432111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12526.571153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12526.575375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12526.584784] LustreError: Skipped 1 previous similar message [12526.597784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12526.609423] Lustre: Skipped 1 previous similar message [12530.975864] Lustre: server umount lustre-OST0000 complete [12536.290145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12536.295153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12536.308793] Lustre: Skipped 1 previous similar message [12536.314249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12541.917093] Lustre: server umount lustre-MDT0000 complete [12546.115808] LustreError: 402909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450573 with bad export cookie 16973835888098815013 [12546.123836] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12546.128751] LustreError: 402909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12546.474865] Lustre: server umount lustre-MDT0001 complete [12555.971599] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12557.786207] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:49:44 (1788450584) [12565.186435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12565.753232] LustreError: 407348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12565.770169] LustreError: 407348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12565.839772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12570.516767] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12571.108175] LustreError: 407349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12576.233858] LustreError: 407348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12580.417350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12580.941942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12586.082063] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12594.794560] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12601.320188] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12608.438722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12608.834324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12615.376521] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12623.719902] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12625.765177] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12628.620466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12636.641448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12636.646983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12636.650424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12636.650432] Lustre: Skipped 1 previous similar message [12636.689417] Lustre: Skipped 2 previous similar messages [12641.159955] Lustre: server umount lustre-OST0000 complete [12646.883599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12646.912505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12646.914566] Lustre: Skipped 1 previous similar message [12651.995634] Lustre: server umount lustre-MDT0000 complete [12652.010558] LustreError: 407349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12652.036463] LustreError: 407349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12655.997962] LustreError: 407329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450683 with bad export cookie 16973835888098815895 [12656.004465] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12656.022640] LustreError: 407329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12656.348146] Lustre: server umount lustre-MDT0001 complete [12669.018557] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [12672.548992] Key type lgssc unregistered [12672.813250] LNet: 411073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12672.822340] LNetError: 411073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12672.843766] LNet: Removed LNI 192.168.204.121@tcp [12673.897227] Key type .llcrypt unregistered [12673.903450] Key type ._llcrypt unregistered [12701.410174] Key type ._llcrypt registered [12701.413768] Key type .llcrypt registered [12701.549090] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [12701.917590] Key type .llcrypt unregistered [12701.926610] Key type ._llcrypt unregistered [12720.418950] Key type ._llcrypt registered [12720.420770] Key type .llcrypt registered [12720.549383] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [12721.965361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12721.995961] alg: No test for adler32 (adler32-zlib) [12723.078485] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12723.355479] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [12725.105123] Key type lgssc registered [12726.666828] Lustre: Echo OBD driver; http://www.lustre.org/ [12735.052900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12735.089769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12736.651971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12741.232544] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12747.241731] LustreError: 413279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12747.265600] LustreError: 413279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12750.585302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12750.971369] LustreError: 413280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12750.993671] LustreError: 413280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12751.002153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12756.452051] LustreError: 413279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12756.466139] LustreError: 413279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12758.320841] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12761.573181] LustreError: 413280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12761.585453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12761.587033] LustreError: 413280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12766.471080] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12766.695130] LustreError: 413279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12766.729723] LustreError: 413279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12773.630235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12773.946370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12778.319556] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12787.172248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12787.184587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12787.201930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12787.723631] Lustre: server umount lustre-OST0000 complete [12793.838793] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12793.853891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12793.867713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12797.004699] Lustre: server umount lustre-MDT0000 complete [12799.459749] LustreError: 413280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12799.480054] LustreError: 413280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12800.797929] LustreError: 413260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450828 with bad export cookie 578794094879202672 [12800.807269] LustreError: 413260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12800.807394] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12811.432276] Lustre: server umount lustre-MDT0001 complete [12822.659809] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [12826.205609] Key type lgssc unregistered [12826.542271] LNet: 416427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12826.558947] LNetError: 416427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12827.623089] LNet: Removed LNI 192.168.204.121@tcp [12828.616527] Key type .llcrypt unregistered [12828.618774] Key type ._llcrypt unregistered [12855.569962] Key type ._llcrypt registered [12855.573232] Key type .llcrypt registered [12855.686164] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [12856.500986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12856.588045] alg: No test for adler32 (adler32-zlib) [12857.806519] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12858.057691] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [12859.744170] Key type lgssc registered [12860.915733] Lustre: Echo OBD driver; http://www.lustre.org/ [12868.758467] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12868.777768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12884.387569] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12889.504390] LustreError: 417817:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 4: rc = -110 [12916.137611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12922.252609] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12930.171797] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12936.852430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12937.422772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12937.515125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12941.270806] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12942.818300] LustreError: 417843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12942.837930] LustreError: 417843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12947.939379] LustreError: 417842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12950.264053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12950.839850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12955.906279] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12964.482476] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12971.821107] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12981.729897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12981.753363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12981.795123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12982.756678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12982.790345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12986.786269] Lustre: server umount lustre-OST0000 complete [12991.969750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12991.988748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12992.017417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12992.027308] Lustre: Skipped 1 previous similar message [12993.706530] Lustre: server umount lustre-MDT0000 complete [12998.113519] LustreError: 418853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12998.124884] LustreError: 418853:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12998.152965] LustreError: 417825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451025 with bad export cookie 2948893419513109975 [12998.155955] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12998.168664] LustreError: 417825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12998.769875] Lustre: server umount lustre-MDT0001 complete [13012.851528] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [13015.869763] Key type lgssc unregistered [13016.171303] LNet: 421386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13016.179173] LNetError: 421386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13016.200657] LNet: Removed LNI 192.168.204.121@tcp [13017.105940] Key type .llcrypt unregistered [13017.108384] Key type ._llcrypt unregistered [13043.761649] Key type ._llcrypt registered [13043.764620] Key type .llcrypt registered [13043.942842] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [13045.154226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13045.186703] alg: No test for adler32 (adler32-zlib) [13046.311061] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13046.552493] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [13048.216147] Key type lgssc registered [13049.390245] Lustre: Echo OBD driver; http://www.lustre.org/ [13058.571921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13058.602812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13060.200632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13065.402203] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13069.797843] LustreError: 422801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13069.809533] LustreError: 422801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13074.761070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13074.932180] LustreError: 422802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13075.145903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13080.360622] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13088.490877] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13094.685757] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13102.194673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13102.619797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13104.708765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13108.671324] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13116.895729] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13121.510506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13121.529630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13121.562166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13124.361200] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.21@tcp (stopping) [13125.610390] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13125.625063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13125.639500] Lustre: Skipped 2 previous similar messages [13127.442505] Lustre: server umount lustre-MDT0000 complete [13129.444608] LustreError: 425243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13130.088233] LustreError: 422783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.21@tcp arrived at 1788451157 with bad export cookie 8053144808813716333 [13130.104873] LustreError: 422783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13131.940176] LustreError: 422783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451159 with bad export cookie 8053144808813715444 [13131.942429] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13131.957925] LustreError: 422783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13135.841542] LustreError: 422802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13135.842621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13135.865775] LustreError: 422802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13135.875085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13135.875094] Lustre: Skipped 1 previous similar message [13135.877645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13138.129778] Lustre: server umount lustre-OST0000 complete [13141.986505] LustreError: 422797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13142.005227] LustreError: 422797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13145.651600] Lustre: server umount lustre-MDT0001 complete [13157.002903] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [13159.901285] Key type lgssc unregistered [13160.220552] LNet: 426484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13160.226551] LNetError: 426484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13160.242191] LNet: Removed LNI 192.168.204.121@tcp [13161.106553] Key type .llcrypt unregistered [13161.109170] Key type ._llcrypt unregistered [13189.281477] Key type ._llcrypt registered [13189.284411] Key type .llcrypt registered [13189.409715] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [13191.191564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13191.246565] alg: No test for adler32 (adler32-zlib) [13192.340460] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13192.683097] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [13194.488685] Key type lgssc registered [13195.770981] Lustre: Echo OBD driver; http://www.lustre.org/ [13205.405543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13205.420828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13206.897593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13211.870486] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13217.254693] LustreError: 427900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13217.276528] LustreError: 427900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13222.000696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13222.308143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13226.784828] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13235.132712] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13241.710810] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13249.447566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13249.789769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13251.832549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13256.860404] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13265.993387] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13272.554853] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13272.569770] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13272.579151] Lustre: Skipped 1 previous similar message [13274.368782] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.21@tcp (stopping) [13277.676488] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13277.688792] Lustre: Skipped 1 previous similar message [13277.919377] Lustre: server umount lustre-MDT0001 complete [13279.461119] LustreError: 427897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13282.793578] LustreError: 430110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13282.818151] LustreError: 430110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13287.908291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13287.909821] LustreError: 428628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13287.927839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13287.962174] Lustre: Skipped 1 previous similar message [13288.033223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13291.257944] Lustre: server umount lustre-OST0000 complete [13293.026929] LustreError: 427900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13296.006064] Lustre: server umount lustre-MDT0000 complete [13313.119675] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [13316.927791] Key type lgssc unregistered [13317.245770] LNet: 431594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13317.258671] LNetError: 431594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13317.280965] LNet: Removed LNI 192.168.204.121@tcp [13318.386797] Key type .llcrypt unregistered [13318.392382] Key type ._llcrypt unregistered [13347.425592] Key type ._llcrypt registered [13347.428684] Key type .llcrypt registered [13347.563706] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [13363.494916] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [13364.693433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13364.910224] alg: No test for adler32 (adler32-zlib) [13366.024502] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13366.286964] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [13368.024259] Key type lgssc registered [13369.254651] Lustre: Echo OBD driver; http://www.lustre.org/ [13416.603780] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [13435.625401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13435.670599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13437.037177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13437.099456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13437.187818] Lustre: lustre-MDT0000: new disk, initializing [13437.337179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13437.389615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13442.213887] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13459.125957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13459.396575] Lustre: lustre-OST0000: new disk, initializing [13459.404111] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13459.410518] Lustre: Skipped 1 previous similar message [13459.421443] Lustre: 436655:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13459.539508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13466.225116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13466.237077] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13466.261726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13466.655384] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13477.441932] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13495.065517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13495.255263] Lustre: 435636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13495.347178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13495.444158] Lustre: lustre-MDT0001: new disk, initializing [13495.627514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13495.690608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13495.711850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13501.359849] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13511.854105] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13512.138905] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13517.585043] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13517.862459] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13526.255671] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.21@tcp (stopping) [13527.527123] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13527.537975] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13527.592447] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13527.596177] Lustre: Skipped 1 previous similar message [13528.872763] Lustre: server umount lustre-MDT0001 complete [13537.765882] LustreError: 435647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13537.786598] LustreError: 435647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13542.886093] LustreError: 435642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13542.899134] LustreError: 435642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13548.003402] LustreError: 435648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13548.041400] LustreError: 435648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13553.130711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13553.134888] LustreError: 435647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13553.145300] Lustre: Skipped 1 previous similar message [13553.146708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13558.252771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13558.258132] LustreError: 435648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13563.371588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13564.384202] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13564.624300] Lustre: server umount lustre-OST0000 complete [13568.482934] LustreError: 435648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13568.505625] LustreError: 435648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13583.329156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13583.835724] Lustre: server umount lustre-MDT0000 complete [13600.122899] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [13603.490849] Key type lgssc unregistered [13603.830841] LNet: 440414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13603.838118] LNetError: 440414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13603.865415] LNet: Removed LNI 192.168.204.121@tcp [13605.036169] Key type .llcrypt unregistered [13605.042521] Key type ._llcrypt unregistered [13654.104459] Key type ._llcrypt registered [13654.106763] Key type .llcrypt registered [13654.192158] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [13654.481720] Key type .llcrypt unregistered [13654.486301] Key type ._llcrypt unregistered [13676.379225] Key type ._llcrypt registered [13676.385394] Key type .llcrypt registered [13676.527668] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [13678.022885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13678.059197] alg: No test for adler32 (adler32-zlib) [13679.403794] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13679.740375] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [13681.464204] Key type lgssc registered [13683.041641] Lustre: Echo OBD driver; http://www.lustre.org/ [13714.833256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13714.847078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13716.055994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13716.091954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13716.206755] Lustre: lustre-MDT0000: new disk, initializing [13716.272520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13716.290606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13720.517169] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13734.584944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13734.682574] Lustre: 443787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13734.763232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13734.770393] Lustre: Skipped 1 previous similar message [13734.878319] Lustre: lustre-MDT0001: new disk, initializing [13734.995481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13735.031245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13735.039369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13741.097727] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13752.580617] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13760.173186] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13768.102650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13768.471238] Lustre: lustre-OST0000: new disk, initializing [13768.480354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13768.495964] Lustre: 445757:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13768.637527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13775.459717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13775.476748] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13775.604916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13776.043633] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13787.020317] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13796.331307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13796.332227] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13796.349642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13796.370505] Lustre: Skipped 1 previous similar message [13801.470377] Lustre: server umount lustre-MDT0000 complete [13805.469446] LustreError: 443779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451833 with bad export cookie 2361809329671063897 [13805.471970] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13805.479596] LustreError: 443779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13805.990468] Lustre: server umount lustre-MDT0001 complete [13820.337476] Lustre: server umount lustre-OST0000 complete [13833.287292] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:10:59 (1788451859) [13841.275310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13841.659947] LustreError: 447787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13841.680273] LustreError: 447787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13841.738480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13846.333324] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13847.009538] LustreError: 447788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13852.130035] LustreError: 447787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13855.606315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13855.897973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13860.945849] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13870.244040] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13877.086190] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13897.529964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13897.769096] Lustre: Found index 0 for lustre-OST0000, updating log [13897.916534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13905.087268] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13913.275505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13914.826878] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13923.814042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13923.824699] Lustre: Skipped 1 previous similar message [13923.830515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13923.831326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13923.841762] Lustre: Skipped 3 previous similar messages [13924.910472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13924.914500] Lustre: Skipped 1 previous similar message [13928.421786] Lustre: server umount lustre-OST0000 complete [13946.338353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13946.340449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13946.367838] Lustre: Skipped 2 previous similar messages [13946.379900] Lustre: Skipped 1 previous similar message [13950.505167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13950.512052] Lustre: Skipped 1 previous similar message [13955.233173] LNet: 1 local NIs in recovery (showing 1): 192.168.204.121@tcp [13959.136416] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13959.265477] Lustre: server umount lustre-MDT0000 complete [13961.698522] LustreError: 447783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13961.727911] LustreError: 447783:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13964.939639] LustreError: 447768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451992 with bad export cookie 2361809329671065290 [13964.944693] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13964.968680] LustreError: 447768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13965.772378] Lustre: server umount lustre-MDT0001 complete [13977.203374] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:13:23 (1788452003) [14604.530924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14605.107899] LustreError: 454204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14605.220583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14610.418098] LustreError: 454205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14610.606092] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [14614.571563] LustreError: 454205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14620.657553] LustreError: 454204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14620.700888] LustreError: 454204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14622.011793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14622.625025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14629.166408] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [14638.718550] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14646.374611] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14660.769697] LNet: 1 local NIs in recovery (showing 1): 192.168.204.121@tcp [14668.284133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14668.681830] Lustre: Found index 0 for lustre-OST0000, updating log [14669.368462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14678.753460] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [14689.436576] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14693.111948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14693.135057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14693.144690] Lustre: Skipped 4 previous similar messages [14698.522898] Lustre: server umount lustre-OST0000 complete [14703.915607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14703.923708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14703.932318] Lustre: Skipped 1 previous similar message [14706.019770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14706.033400] Lustre: Skipped 4 previous similar messages [14709.307143] Lustre: server umount lustre-MDT0000 complete [14710.113649] LustreError: 454205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14710.134976] LustreError: 454205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14713.510906] LustreError: 454186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452741 with bad export cookie 2361809329671066375 [14713.518670] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14713.531438] LustreError: 454186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14714.013784] Lustre: server umount lustre-MDT0001 complete [14721.184656] LNet: 1 local NIs in recovery (showing 1): 192.168.204.121@tcp [14747.615796] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [14759.156147] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [14805.511704] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:27:10 (1788452830) [14826.975817] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [14829.715914] Key type lgssc unregistered [14830.085473] LNet: 460825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14830.095996] LNetError: 460825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14931.501828] LNet: Removed LNI 192.168.204.121@tcp [14932.362637] Key type .llcrypt unregistered [14932.365332] Key type ._llcrypt unregistered [14949.624665] Key type ._llcrypt registered [14949.630906] Key type .llcrypt registered [14949.794322] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [14950.787680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14950.830067] alg: No test for adler32 (adler32-zlib) [14951.974150] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14952.191091] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [14953.896139] Key type lgssc registered [14955.174759] Lustre: Echo OBD driver; http://www.lustre.org/ [14960.244991] LNet: Added LNI 192.168.204.121@tcp100 [8/256/0/180] [14961.164363] LNet: Added LNI 192.168.204.121@tcp101 [8/256/0/180] [14962.166797] LNet: Added LNI 192.168.204.121@tcp102 [8/256/0/180] [14963.164831] LNet: Added LNI 192.168.204.121@tcp103 [8/256/0/180] [14964.179935] LNet: Added LNI 192.168.204.121@tcp104 [8/256/0/180] [14965.078613] LNet: Added LNI 192.168.204.121@tcp105 [8/256/0/180] [14966.150928] LNet: Added LNI 192.168.204.121@tcp106 [8/256/0/180] [14967.102050] LNet: Added LNI 192.168.204.121@tcp107 [8/256/0/180] [14968.112462] LNet: Added LNI 192.168.204.121@tcp108 [8/256/0/180] [14968.931843] LNet: Added LNI 192.168.204.121@tcp109 [8/256/0/180] [14969.831331] LNet: Added LNI 192.168.204.121@tcp110 [8/256/0/180] [14970.807477] LNet: Added LNI 192.168.204.121@tcp111 [8/256/0/180] [14971.747702] LNet: Added LNI 192.168.204.121@tcp112 [8/256/0/180] [14972.686314] LNet: Added LNI 192.168.204.121@tcp113 [8/256/0/180] [14973.629193] LNet: Added LNI 192.168.204.121@tcp114 [8/256/0/180] [14974.598704] LNet: Added LNI 192.168.204.121@tcp115 [8/256/0/180] [14975.663723] LNet: Added LNI 192.168.204.121@tcp116 [8/256/0/180] [14976.589267] LNet: Added LNI 192.168.204.121@tcp117 [8/256/0/180] [14977.574824] LNet: Added LNI 192.168.204.121@tcp118 [8/256/0/180] [14978.380696] LNet: Added LNI 192.168.204.121@tcp119 [8/256/0/180] [14979.266460] LNet: Added LNI 192.168.204.121@tcp120 [8/256/0/180] [14980.129424] LNet: Added LNI 192.168.204.121@tcp121 [8/256/0/180] [14980.930241] LNet: Added LNI 192.168.204.121@tcp122 [8/256/0/180] [14981.853354] LNet: Added LNI 192.168.204.121@tcp123 [8/256/0/180] [14982.914266] LNet: Added LNI 192.168.204.121@tcp124 [8/256/0/180] [14983.775545] LNet: Added LNI 192.168.204.121@tcp125 [8/256/0/180] [14984.690730] LNet: Added LNI 192.168.204.121@tcp126 [8/256/0/180] [14985.523141] LNet: Added LNI 192.168.204.121@tcp127 [8/256/0/180] [14986.357348] LNet: Added LNI 192.168.204.121@tcp128 [8/256/0/180] [14987.221629] LNet: Added LNI 192.168.204.121@tcp129 [8/256/0/180] [14988.074721] LNet: Added LNI 192.168.204.121@tcp130 [8/256/0/180] [14988.983866] LNet: Added LNI 192.168.204.121@tcp131 [8/256/0/180] [14989.953566] LNet: Added LNI 192.168.204.121@tcp132 [8/256/0/180] [14990.863140] LNet: Added LNI 192.168.204.121@tcp133 [8/256/0/180] [14991.642757] LNet: Added LNI 192.168.204.121@tcp134 [8/256/0/180] [14992.536311] LNet: Added LNI 192.168.204.121@tcp135 [8/256/0/180] [14998.932443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14998.966751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15000.151653] Lustre: 463853:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.121@tcp: can't add NID 192.168.204.121@tcp130: rc = -75 [15000.266832] LNetError: 463873:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.121 [15000.274246] LNetError: Protocol error connecting to 192.168.204.121@tcp116 at host 192.168.204.121:988: is it running a compatible version of Lustre? [15000.285525] LNetError: Skipped 2 previous similar messages [15000.352902] LNet: 8 local NIs in recovery (showing 8): 192.168.204.121@tcp120, 192.168.204.121@tcp121, 192.168.204.121@tcp123, 192.168.204.121@tcp124, 192.168.204.121@tcp127, 192.168.204.121@tcp126, 192.168.204.121@tcp128, 192.168.204.121@tcp129 [15000.430942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15000.451453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15000.517713] Lustre: lustre-MDT0000: new disk, initializing [15000.636881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15000.669862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15001.381498] LNetError: 463876:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.121 [15001.394488] LNetError: 463876:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [15001.411222] LNetError: Protocol error connecting to 192.168.204.121@tcp124 at host 192.168.204.121:988: is it running a compatible version of Lustre? [15001.426533] LNetError: Skipped 11 previous similar messages [15003.425193] LNetError: 461309:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.121 [15003.433605] LNetError: 461309:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [15003.438446] LNetError: Protocol error connecting to 192.168.204.121@tcp124 at host 192.168.204.121:988: is it running a compatible version of Lustre? [15004.813157] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [15007.521805] LNetError: 463878:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.121 [15007.537546] LNetError: Protocol error connecting to 192.168.204.121@tcp124 at host 192.168.204.121:988: is it running a compatible version of Lustre? [15015.729036] LNetError: 463880:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.121 [15015.739949] LNetError: Protocol error connecting to 192.168.204.121@tcp124 at host 192.168.204.121:988: is it running a compatible version of Lustre? [15016.180711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15016.417374] Lustre: 463871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15016.520747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15016.529053] Lustre: Skipped 1 previous similar message [15016.594959] Lustre: lustre-MDT0001: new disk, initializing [15016.631532] Lustre: 464670:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.121@tcp: can't add NID 192.168.204.121@tcp130: rc = -75 [15016.639282] Lustre: 464670:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15016.736979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15016.771699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15016.779993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15020.991044] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [15025.687497] Lustre: 465095:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.121@tcp: can't add NID 192.168.204.121@tcp130: rc = -75 [15025.693720] Lustre: 465095:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15030.101512] LNetError: 463873:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.121 [15030.129970] LNetError: Protocol error connecting to 192.168.204.121@tcp125 at host 192.168.204.121:988: is it running a compatible version of Lustre? [15035.114406] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15045.632915] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15046.442359] LNetError: 463873:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.121 [15046.477634] LNetError: 463873:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [15046.496495] LNetError: Protocol error connecting to 192.168.204.121@tcp128 at host 192.168.204.121:988: is it running a compatible version of Lustre? [15046.521596] LNetError: Skipped 9 previous similar messages [15055.609787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15056.091397] Lustre: lustre-OST0000: new disk, initializing [15056.115255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15056.129143] Lustre: 465846:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15056.248929] Lustre: 465851:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.121@tcp: can't add NID 192.168.204.121@tcp130: rc = -75 [15056.262689] Lustre: 465851:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15056.317400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15060.772425] LNet: 1 local NIs in recovery (showing 1): 192.168.204.121@tcp124 [15065.640603] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [15065.721903] Lustre: 466199:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.121@tcp: can't add NID 192.168.204.121@tcp130: rc = -75 [15065.754051] Lustre: 466199:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15065.887448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15065.919142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15066.205542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15076.303833] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15086.561847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15086.578829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15086.594789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15086.596729] Lustre: Skipped 1 previous similar message [15086.631430] Lustre: Skipped 1 previous similar message [15091.789996] Lustre: server umount lustre-OST0000 complete [15094.572100] LNetError: 463885:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.121 [15094.586485] LNetError: 463885:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 23 previous similar messages [15094.592135] LNetError: Protocol error connecting to 192.168.204.121@tcp125 at host 192.168.204.121:988: is it running a compatible version of Lustre? [15094.607599] LNetError: Skipped 23 previous similar messages [15096.806405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15096.822669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15096.827893] Lustre: Skipped 1 previous similar message [15101.922280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15101.938984] Lustre: Skipped 1 previous similar message [15102.886161] Lustre: server umount lustre-MDT0000 complete [15107.048061] LustreError: 465099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15107.070337] LustreError: 465099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15107.285147] LustreError: 463863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453134 with bad export cookie 1631306354325978623 [15107.287864] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15107.296206] LustreError: 463863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15107.840330] Lustre: server umount lustre-MDT0001 complete [15119.503075] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [15122.807260] Key type lgssc unregistered [15123.293160] LNet: 467759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15123.303112] LNetError: 467759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15123.313535] LNet: Removed LNI 192.168.204.121@tcp [15124.324059] LNet: Removed LNI 192.168.204.121@tcp100 [15124.330580] LNetError: 467759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15124.343509] LNetError: 467759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15124.355596] LNet: Removed LNI 192.168.204.121@tcp101 [15125.408549] LNet: Removed LNI 192.168.204.121@tcp102 [15125.416105] LNetError: 467759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15125.431359] LNetError: 467759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15125.444163] LNet: Removed LNI 192.168.204.121@tcp103 [15126.496580] LNet: Removed LNI 192.168.204.121@tcp104 [15127.584213] LNet: Removed LNI 192.168.204.121@tcp105 [15127.595963] LNetError: 467759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15127.613833] LNetError: 467759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15127.630721] LNet: Removed LNI 192.168.204.121@tcp106 [15128.672147] LNet: Removed LNI 192.168.204.121@tcp107 [15129.697925] LNet: Removed LNI 192.168.204.121@tcp108 [15130.720127] LNet: Removed LNI 192.168.204.121@tcp109 [15131.744154] LNet: Removed LNI 192.168.204.121@tcp110 [15131.746173] LNetError: 467759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15131.754043] LNetError: 467759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15131.771147] LNet: Removed LNI 192.168.204.121@tcp111 [15132.832143] LNet: Removed LNI 192.168.204.121@tcp112 [15133.857196] LNet: Removed LNI 192.168.204.121@tcp113 [15134.880261] LNet: Removed LNI 192.168.204.121@tcp114 [15135.906508] LNet: Removed LNI 192.168.204.121@tcp115 [15136.928523] LNet: Removed LNI 192.168.204.121@tcp116 [15137.952359] LNet: Removed LNI 192.168.204.121@tcp117 [15138.976165] LNet: Removed LNI 192.168.204.121@tcp118 [15140.000186] LNet: Removed LNI 192.168.204.121@tcp119 [15140.010399] LNetError: 467759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15140.018131] LNetError: 467759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15141.088194] LNet: Removed LNI 192.168.204.121@tcp120 [15142.112202] LNet: Removed LNI 192.168.204.121@tcp121 [15143.138177] LNet: Removed LNI 192.168.204.121@tcp122 [15144.160309] LNet: Removed LNI 192.168.204.121@tcp123 [15144.165174] LNet: Removed LNI 192.168.204.121@tcp124 [15145.184308] LNet: Removed LNI 192.168.204.121@tcp125 [15146.272279] LNet: Removed LNI 192.168.204.121@tcp126 [15147.296594] LNet: Removed LNI 192.168.204.121@tcp127 [15148.322435] LNet: Removed LNI 192.168.204.121@tcp128 [15149.344443] LNet: Removed LNI 192.168.204.121@tcp129 [15149.347277] LNet: Removed LNI 192.168.204.121@tcp130 [15149.350673] LNet: Removed LNI 192.168.204.121@tcp131 [15149.354158] LNet: Removed LNI 192.168.204.121@tcp132 [15149.358992] LNet: Removed LNI 192.168.204.121@tcp133 [15149.367070] LNet: Removed LNI 192.168.204.121@tcp134 [15149.379187] LNet: Removed LNI 192.168.204.121@tcp135 [15150.509109] Key type .llcrypt unregistered [15150.515879] Key type ._llcrypt unregistered [15181.697263] Key type ._llcrypt registered [15181.700285] Key type .llcrypt registered [15181.828809] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [15198.748062] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [15200.028786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15200.157678] alg: No test for adler32 (adler32-zlib) [15201.375882] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15201.573657] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [15203.288753] Key type lgssc registered [15204.759553] Lustre: Echo OBD driver; http://www.lustre.org/ [15248.378514] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:34:34 (1788453274) [15267.296582] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [15270.344324] Key type lgssc unregistered [15270.680540] LNet: 471847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15270.689318] LNetError: 471847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15270.725745] LNet: Removed LNI 192.168.204.121@tcp [15271.833490] Key type .llcrypt unregistered [15271.835315] Key type ._llcrypt unregistered [15290.753392] Key type ._llcrypt registered [15290.756422] Key type .llcrypt registered [15290.863682] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [15292.011514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15292.162850] alg: No test for adler32 (adler32-zlib) [15293.337388] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15293.583382] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [15295.256172] Key type lgssc registered [15296.644581] Lustre: Echo OBD driver; http://www.lustre.org/ [15307.498458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15307.521108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15308.795312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15308.861722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15308.977255] Lustre: lustre-MDT0000: new disk, initializing [15309.107421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15309.132220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15314.284558] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [15326.927242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15327.159789] Lustre: 473062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15327.244113] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15327.249090] Lustre: Skipped 1 previous similar message [15327.389199] Lustre: lustre-MDT0001: new disk, initializing [15327.520832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15327.560157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15327.575526] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15333.198971] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [15343.952258] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15350.799887] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15359.321847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15359.756537] Lustre: lustre-OST0000: new disk, initializing [15359.765274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15359.776952] Lustre: 475033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15359.893643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15367.800200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15367.814143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15367.977229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15369.142695] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [15380.564495] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15386.096243] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing lnet_if_list [15392.968919] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15393.028390] LNet: Added LNI 192.168.204.121@tcp10 [8/256/0/180] [15399.852379] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15399.882318] LNet: Added LNI 192.168.204.121@tcp11 [8/256/0/180] [15405.440415] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15405.512689] LNet: Added LNI 192.168.204.121@tcp12 [8/256/0/180] [15412.963105] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15413.040492] LNet: Added LNI 192.168.204.121@tcp13 [8/256/0/180] [15421.221258] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15421.324892] LNet: Added LNI 192.168.204.121@tcp14 [8/256/0/180] [15426.867492] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15426.887683] LNet: Added LNI 192.168.204.121@tcp15 [8/256/0/180] [15432.635338] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15432.716509] LNet: Added LNI 192.168.204.121@tcp16 [8/256/0/180] [15437.782881] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15437.835351] LNet: Added LNI 192.168.204.121@tcp17 [8/256/0/180] [15443.679837] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15443.709415] LNet: Added LNI 192.168.204.121@tcp18 [8/256/0/180] [15450.005849] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15450.041182] LNet: Added LNI 192.168.204.121@tcp19 [8/256/0/180] [15456.585522] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15456.609466] LNet: Added LNI 192.168.204.121@tcp20 [8/256/0/180] [15463.264185] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15463.299484] LNet: Added LNI 192.168.204.121@tcp21 [8/256/0/180] [15471.844867] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15471.892504] LNet: Added LNI 192.168.204.121@tcp22 [8/256/0/180] [15479.249345] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15479.283252] LNet: Added LNI 192.168.204.121@tcp23 [8/256/0/180] [15486.357738] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15486.413177] LNet: Added LNI 192.168.204.121@tcp24 [8/256/0/180] [15493.466975] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15493.553293] LNet: Added LNI 192.168.204.121@tcp25 [8/256/0/180] [15502.559119] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15502.620884] LNet: Added LNI 192.168.204.121@tcp26 [8/256/0/180] [15509.861254] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15509.898586] LNet: Added LNI 192.168.204.121@tcp27 [8/256/0/180] [15517.450721] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15517.517925] LNet: Added LNI 192.168.204.121@tcp28 [8/256/0/180] [15525.255351] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15525.282887] LNet: Added LNI 192.168.204.121@tcp29 [8/256/0/180] [15533.480811] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15533.527088] LNet: Added LNI 192.168.204.121@tcp30 [8/256/0/180] [15540.674518] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15540.724529] LNet: Added LNI 192.168.204.121@tcp31 [8/256/0/180] [15548.968599] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15548.995773] LNet: Added LNI 192.168.204.121@tcp32 [8/256/0/180] [15556.841081] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15556.885895] LNet: Added LNI 192.168.204.121@tcp33 [8/256/0/180] [15564.837908] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15564.869898] LNet: Added LNI 192.168.204.121@tcp34 [8/256/0/180] [15571.737248] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15571.760847] LNet: Added LNI 192.168.204.121@tcp35 [8/256/0/180] [15582.534738] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15582.582924] LNet: Added LNI 192.168.204.121@tcp36 [8/256/0/180] [15591.804384] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15591.856028] LNet: Added LNI 192.168.204.121@tcp37 [8/256/0/180] [15600.599311] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15600.636062] LNet: Added LNI 192.168.204.121@tcp38 [8/256/0/180] [15608.218091] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15608.262680] LNet: Added LNI 192.168.204.121@tcp39 [8/256/0/180] [15616.019745] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15616.039294] LNet: Added LNI 192.168.204.121@tcp40 [8/256/0/180] [15625.229400] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15625.280859] LNet: Added LNI 192.168.204.121@tcp41 [8/256/0/180] [15636.070262] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15636.094450] LNet: Added LNI 192.168.204.121@tcp42 [8/256/0/180] [15645.503337] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15645.564932] LNet: Added LNI 192.168.204.121@tcp43 [8/256/0/180] [15654.217243] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15654.256623] LNet: Added LNI 192.168.204.121@tcp44 [8/256/0/180] [15663.134212] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15663.192271] LNet: Added LNI 192.168.204.121@tcp45 [8/256/0/180] [15674.055231] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15674.119373] LNet: Added LNI 192.168.204.121@tcp46 [8/256/0/180] [15685.156208] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15685.204275] LNet: Added LNI 192.168.204.121@tcp47 [8/256/0/180] [15694.921697] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15694.946705] LNet: Added LNI 192.168.204.121@tcp48 [8/256/0/180] [15704.362683] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15704.390657] LNet: Added LNI 192.168.204.121@tcp49 [8/256/0/180] [15713.231796] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15713.300175] LNet: Added LNI 192.168.204.121@tcp50 [8/256/0/180] [15734.757544] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15734.760550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15734.776321] Lustre: Skipped 2 previous similar messages [15734.794722] Lustre: Skipped 2 previous similar messages [15739.352894] Lustre: server umount lustre-MDT0000 complete [15743.601694] LustreError: 473054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453771 with bad export cookie 1395091366838780717 [15743.614209] LustreError: 473054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15743.615451] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15744.066055] Lustre: server umount lustre-MDT0001 complete [15760.352197] Lustre: 472333:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453772/real 1788453772] req@00000000546d0fea x1875329229726592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453788 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15760.382668] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15762.749636] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [15764.512244] Lustre: 472333:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453776/real 1788453776] req@00000000da312f95 x1875329229726720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453792 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15769.504327] Lustre: 472335:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453781/real 1788453781] req@000000009c970984 x1875329229726976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15771.778096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15775.651467] Lustre: 472335:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453787/real 1788453787] req@00000000002efd21 x1875329229727104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15789.093740] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x135c5c34e5a3cdde [15789.113261] Lustre: MGC192.168.204.121@tcp: Connection restored to 0@lo (at 0@lo) [15789.328284] LustreError: 475025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15789.353449] LustreError: 475025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15789.394626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15790.444502] LustreError: 475026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15790.449047] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15791.461687] LustreError: 475025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15794.590074] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [15796.710895] LustreError: 475026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15801.829818] LustreError: 475025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15803.988896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15804.540396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15805.567659] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15809.462971] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [15817.919350] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15823.807565] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15845.347885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15845.357242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15845.392840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15845.863345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15846.890885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15846.904800] Lustre: Skipped 2 previous similar messages [15852.024154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15852.041380] Lustre: Skipped 1 previous similar message [15859.168184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15859.387906] Lustre: server umount lustre-OST0000 complete [15865.315680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15865.319746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15865.341947] Lustre: Skipped 2 previous similar messages [15865.364147] Lustre: Skipped 3 previous similar messages [15875.555425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15875.569523] Lustre: Skipped 3 previous similar messages [15878.114074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15878.269796] Lustre: server umount lustre-MDT0000 complete [15880.676578] LustreError: 499289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15880.706456] LustreError: 499289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15883.217910] LustreError: 473055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453910 with bad export cookie 1395091366838783454 [15883.232195] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15883.545697] Lustre: server umount lustre-MDT0001 complete [15896.820420] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [15900.378561] Key type lgssc unregistered [15900.906418] LNet: 502090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15900.921256] LNetError: 502090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15901.985748] LNet: Removed LNI 192.168.204.121@tcp [15901.994214] LNetError: 502090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15902.000785] LNet: Removed LNI 192.168.204.121@tcp10 [15902.004456] LNet: Removed LNI 192.168.204.121@tcp11 [15902.008066] LNet: Removed LNI 192.168.204.121@tcp12 [15902.011490] LNet: Removed LNI 192.168.204.121@tcp13 [15902.015326] LNet: Removed LNI 192.168.204.121@tcp14 [15902.018568] LNet: Removed LNI 192.168.204.121@tcp15 [15902.022891] LNet: Removed LNI 192.168.204.121@tcp16 [15902.026627] LNet: Removed LNI 192.168.204.121@tcp17 [15902.029582] LNet: Removed LNI 192.168.204.121@tcp18 [15902.033802] LNet: Removed LNI 192.168.204.121@tcp19 [15902.038537] LNet: Removed LNI 192.168.204.121@tcp20 [15902.046559] LNet: Removed LNI 192.168.204.121@tcp21 [15902.055552] LNet: Removed LNI 192.168.204.121@tcp22 [15902.064334] LNet: Removed LNI 192.168.204.121@tcp23 [15902.068252] LNet: Removed LNI 192.168.204.121@tcp24 [15902.072836] LNet: Removed LNI 192.168.204.121@tcp25 [15902.089608] LNet: Removed LNI 192.168.204.121@tcp26 [15902.106332] LNet: Removed LNI 192.168.204.121@tcp27 [15902.108345] LNet: Removed LNI 192.168.204.121@tcp28 [15902.129351] LNet: Removed LNI 192.168.204.121@tcp29 [15902.134905] LNet: Removed LNI 192.168.204.121@tcp30 [15902.139256] LNet: Removed LNI 192.168.204.121@tcp31 [15902.144926] LNet: Removed LNI 192.168.204.121@tcp32 [15902.151922] LNet: Removed LNI 192.168.204.121@tcp33 [15902.155290] LNet: Removed LNI 192.168.204.121@tcp34 [15902.161990] LNet: Removed LNI 192.168.204.121@tcp35 [15902.167430] LNet: Removed LNI 192.168.204.121@tcp36 [15902.169329] LNet: Removed LNI 192.168.204.121@tcp37 [15902.176551] LNet: Removed LNI 192.168.204.121@tcp38 [15902.186194] LNet: Removed LNI 192.168.204.121@tcp39 [15902.194434] LNet: Removed LNI 192.168.204.121@tcp40 [15902.202462] LNet: Removed LNI 192.168.204.121@tcp41 [15902.217635] LNet: Removed LNI 192.168.204.121@tcp42 [15902.228851] LNet: Removed LNI 192.168.204.121@tcp43 [15902.235212] LNet: Removed LNI 192.168.204.121@tcp44 [15902.240681] LNet: Removed LNI 192.168.204.121@tcp45 [15902.248425] LNet: Removed LNI 192.168.204.121@tcp46 [15902.255115] LNet: Removed LNI 192.168.204.121@tcp47 [15902.276383] LNet: Removed LNI 192.168.204.121@tcp48 [15902.287172] LNet: Removed LNI 192.168.204.121@tcp49 [15902.310340] LNet: Removed LNI 192.168.204.121@tcp50 [15903.372923] Key type .llcrypt unregistered [15903.376888] Key type ._llcrypt unregistered [15927.266038] Key type ._llcrypt registered [15927.267486] Key type .llcrypt registered [15927.410426] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [15945.457474] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [15946.983532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15947.131515] alg: No test for adler32 (adler32-zlib) [15948.496085] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15948.956878] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [15950.720226] Key type lgssc registered [15952.109732] Lustre: Echo OBD driver; http://www.lustre.org/ [15996.305771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15996.332290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15997.585736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15997.610362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15997.733850] Lustre: lustre-MDT0000: new disk, initializing [15997.894333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15997.930269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16003.893690] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [16015.886367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16016.006451] Lustre: 505282:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16016.179436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16016.185681] Lustre: Skipped 1 previous similar message [16016.303613] Lustre: lustre-MDT0001: new disk, initializing [16016.468029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16016.524641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16016.535576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16022.050795] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [16033.723337] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16042.436278] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16052.757258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16053.114617] Lustre: *** cfs_fail_loc=90f, val=0*** [16053.223239] Lustre: lustre-OST0000: new disk, initializing [16053.227437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16053.235531] Lustre: 507302:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16053.395043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16060.525111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16060.547869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16060.703887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16062.252834] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [16072.818447] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16081.381788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16081.382699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16081.401737] LustreError: Skipped 1 previous similar message [16081.404683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16081.412448] Lustre: Skipped 1 previous similar message [16086.504198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16086.515329] Lustre: Skipped 1 previous similar message [16087.757835] Lustre: server umount lustre-OST0000 complete [16093.154136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16093.163319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16093.174725] Lustre: Skipped 1 previous similar message [16093.177579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16096.738815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16096.742519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16098.126708] Lustre: server umount lustre-MDT0000 complete [16102.803060] LustreError: 505276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454130 with bad export cookie 2263754753317148861 [16102.828253] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16103.155675] Lustre: server umount lustre-MDT0001 complete [16115.656587] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [16119.092051] Key type lgssc unregistered [16119.376858] LNet: 509207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16119.386208] LNetError: 509207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16119.397725] LNet: Removed LNI 192.168.204.121@tcp [16120.510195] Key type .llcrypt unregistered [16120.515882] Key type ._llcrypt unregistered [16145.024922] Key type ._llcrypt registered [16145.027754] Key type .llcrypt registered [16145.177597] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [16146.452362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16146.487927] alg: No test for adler32 (adler32-zlib) [16147.474894] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [16147.668485] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [16149.320185] Key type lgssc registered [16150.911836] Lustre: Echo OBD driver; http://www.lustre.org/ [16160.695076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16160.722222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16162.357613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16168.292872] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [16172.514261] LustreError: 510477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16172.550757] LustreError: 510477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16177.639593] LustreError: 510478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16178.531150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16178.944993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16183.793306] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [16192.890417] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16199.973369] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16209.675114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16210.298275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16218.152669] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [16228.501491] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16244.924464] loop0: detected capacity change from 0 to 204800000 [16245.163840] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16245.175224] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16245.189928] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16252.487219] loop0: detected capacity change from 0 to 204800000 [16252.625895] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16252.649524] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16252.685890] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16259.574383] loop0: detected capacity change from 0 to 204800000 [16266.614421] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16266.779833] Lustre: fs15246-OST0000: new disk, initializing [16266.782508] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16266.786572] Lustre: 514077:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16266.842695] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16274.338791] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [16286.149740] loop1: detected capacity change from 0 to 204800000 [16292.936035] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16293.016258] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16293.069086] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16293.144325] Lustre: fs15246-MDT0000: new disk, initializing [16293.356092] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16293.434857] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16293.616746] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16294.690985] Lustre: 515199:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000008899ac35 x1875330125058048/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788454338 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16299.275389] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [16299.761366] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16299.771943] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16299.796188] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16324.072772] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16324.100622] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16327.664872] Lustre: server umount fs15246-MDT0000 complete [16329.192356] LustreError: 513356:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16334.318445] LustreError: 510474:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16337.181779] Lustre: server umount fs15246-OST0000 complete [16350.693497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16350.699488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16350.720393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16351.717194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16351.734682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16351.764653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16354.787832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16354.795000] Lustre: Skipped 1 previous similar message [16356.925363] Lustre: server umount lustre-OST0000 complete [16363.498281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16363.507224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16363.529769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16367.350866] Lustre: server umount lustre-MDT0000 complete [16370.151628] LustreError: 510472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16370.174573] LustreError: 510472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16371.477279] LustreError: 510459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454399 with bad export cookie 8997786443373786863 [16371.501632] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16371.883627] Lustre: server umount lustre-MDT0001 complete [16384.970622] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [16388.093831] Key type lgssc unregistered [16388.438933] LNet: 518295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16388.458759] LNetError: 518295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16388.491601] LNet: Removed LNI 192.168.204.121@tcp [16389.643600] Key type .llcrypt unregistered [16389.646304] Key type ._llcrypt unregistered [16411.459680] Key type ._llcrypt registered [16411.461842] Key type .llcrypt registered [16411.572439] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [16413.035773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16413.192655] alg: No test for adler32 (adler32-zlib) [16414.262898] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [16414.497476] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [16416.249164] Key type lgssc registered [16417.724209] Lustre: Echo OBD driver; http://www.lustre.org/ [16460.963219] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [16472.632742] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [16516.169645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16516.190973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16517.534903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16517.678419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16517.773951] Lustre: lustre-MDT0000: new disk, initializing [16517.892233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16517.940377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16522.518382] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [16534.509312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16534.698934] Lustre: 522762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16534.779462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16534.799377] Lustre: Skipped 1 previous similar message [16534.995993] Lustre: lustre-MDT0001: new disk, initializing [16535.144854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16535.186886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16535.198886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16539.745625] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [16552.726552] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16560.607689] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16568.335328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16568.752199] Lustre: lustre-OST0000: new disk, initializing [16568.763954] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16568.785751] Lustre: 524731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16568.933671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16577.018702] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [16577.144404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16577.175190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16577.391802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16589.190437] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16595.065336] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16595.470424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16600.847426] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16601.082757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16603.108939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16603.110892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16603.113255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16603.117643] LustreError: Skipped 1 previous similar message [16603.125684] Lustre: Skipped 1 previous similar message [16608.228927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16608.239384] Lustre: Skipped 2 previous similar messages [16609.237961] Lustre: server umount lustre-OST0000 complete [16616.931899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16616.941514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16616.961661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16618.467063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16620.236762] Lustre: server umount lustre-MDT0000 complete [16625.612414] LustreError: 522754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454653 with bad export cookie 11889414819374066621 [16625.615902] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16625.642685] LustreError: 522754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16626.347687] Lustre: server umount lustre-MDT0001 complete [16660.944607] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [16673.997831] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [16713.340887] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16427 sec ========= 12:58:59 (1788454739) [16715.477362] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:59:01 (1788454741) === [16719.876920] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:59:06 (1788454746) === [16768.608479] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [16772.680896] Key type lgssc unregistered [16773.023270] LNet: 531379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16773.041536] LNetError: 531379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16773.070699] LNet: Removed LNI 192.168.204.121@tcp [16774.162848] Key type .llcrypt unregistered [16774.165639] Key type ._llcrypt unregistered