[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490994084 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2592MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003362] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.008270] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.012023] LSM: Security Framework initializing [ 0.013057] Yama: becoming mindful. [ 0.014059] SELinux: Initializing. [ 0.015053] *** VALIDATE selinux *** [ 0.024253] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029333] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032116] *** VALIDATE tmpfs *** [ 0.034436] *** VALIDATE proc *** [ 0.035270] *** VALIDATE cgroup *** [ 0.036018] *** VALIDATE cgroup2 *** [ 0.037292] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038189] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040030] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.044546] debug: unmapping init [mem 0xffffffff84859000-0xffffffff84860fff] [ 0.046208] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047765] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.049014] ... bit width: 48 [ 0.050015] ... generic registers: 4 [ 0.051015] ... value mask: 0000ffffffffffff [ 0.052013] ... max period: 00007fffffffffff [ 0.053013] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055308] rcu: Hierarchical SRCU implementation. [ 0.057558] smp: Bringing up secondary CPUs ... [ 0.058636] x86: Booting SMP configuration: [ 0.059032] .... node #0, CPUs: #1 #2 #3 [ 0.063011] smp: Brought up 1 node, 4 CPUs [ 0.065020] smpboot: Max logical packages: 1 [ 0.066014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.200468] node 0 deferred pages initialised in 132ms [ 0.203134] devtmpfs: initialized [ 0.204261] x86/mm: Memory block size: 128MB [ 0.206717] gcov: version magic: 0x41383552 [ 0.207672] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.211122] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.213346] pinctrl core: initialized pinctrl subsystem [ 0.216197] [ 0.216821] ************************************************************* [ 0.219016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.222016] ** ** [ 0.224013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.226016] ** ** [ 0.229016] ** This means that this kernel is built to expose internal ** [ 0.231014] ** IOMMU data structures, which may compromise security on ** [ 0.234014] ** your system. ** [ 0.236013] ** ** [ 0.238016] ** If you see this message and you are not debugging the ** [ 0.240015] ** kernel, report this immediately to your vendor! ** [ 0.243011] ** ** [ 0.245014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.247047] ************************************************************* [ 0.250701] NET: Registered protocol family 16 [ 0.252452] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.255067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.258067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.263091] cpuidle: using governor menu [ 0.265000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.267954] PCI: Using configuration type 1 for base access [ 0.270171] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.282198] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.285129] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.288089] cryptd: max_cpu_qlen set to 1000 [ 0.289319] ACPI: Added _OSI(Module Device) [ 0.291027] ACPI: Added _OSI(Processor Device) [ 0.293023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.295103] ACPI: Added _OSI(Processor Aggregator Device) [ 0.302045] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.310701] ACPI: Interpreter enabled [ 0.312093] ACPI: PM: (supports S0 S3 S4 S5) [ 0.314018] ACPI: Using IOAPIC for interrupt routing [ 0.316173] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.320549] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.331245] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.334064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.336022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.340096] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.344441] acpiphp: Slot [2] registered [ 0.346134] acpiphp: Slot [5] registered [ 0.347138] acpiphp: Slot [6] registered [ 0.349138] acpiphp: Slot [7] registered [ 0.350126] acpiphp: Slot [8] registered [ 0.352140] acpiphp: Slot [9] registered [ 0.353143] acpiphp: Slot [10] registered [ 0.355116] acpiphp: Slot [3] registered [ 0.356111] acpiphp: Slot [4] registered [ 0.358124] acpiphp: Slot [11] registered [ 0.359122] acpiphp: Slot [12] registered [ 0.361135] acpiphp: Slot [13] registered [ 0.362122] acpiphp: Slot [14] registered [ 0.364131] acpiphp: Slot [15] registered [ 0.366169] acpiphp: Slot [16] registered [ 0.368202] acpiphp: Slot [17] registered [ 0.369161] acpiphp: Slot [18] registered [ 0.371168] acpiphp: Slot [19] registered [ 0.373200] acpiphp: Slot [20] registered [ 0.375164] acpiphp: Slot [21] registered [ 0.376152] acpiphp: Slot [22] registered [ 0.378177] acpiphp: Slot [23] registered [ 0.380136] acpiphp: Slot [24] registered [ 0.381148] acpiphp: Slot [25] registered [ 0.383191] acpiphp: Slot [26] registered [ 0.385170] acpiphp: Slot [27] registered [ 0.386161] acpiphp: Slot [28] registered [ 0.388163] acpiphp: Slot [29] registered [ 0.389179] acpiphp: Slot [30] registered [ 0.391148] acpiphp: Slot [31] registered [ 0.393092] PCI host bridge to bus 0000:00 [ 0.394031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.397048] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.399049] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.402033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.405026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.408031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.409201] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.413196] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.416513] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.427021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.431563] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.435021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.437019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.441021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.443382] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.446875] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.450069] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.453989] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.459029] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.476030] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.481016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.487059] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.494013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.501018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.518021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.529172] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.535018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.543017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.562017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.576117] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.583023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.589019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.609018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.620290] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.629013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.635018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.653021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.664000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.669016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.674019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.698017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.709000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.718017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.731019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.747034] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.761150] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.763365] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.765965] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.768390] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.771245] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.775322] iommu: Default domain type: Passthrough [ 0.777776] SCSI subsystem initialized [ 0.778150] ACPI: bus type USB registered [ 0.780135] usbcore: registered new interface driver usbfs [ 0.782084] usbcore: registered new interface driver hub [ 0.783128] usbcore: registered new device driver usb [ 0.785277] pps_core: LinuxPPS API ver. 1 registered [ 0.788014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.791073] PTP clock support registered [ 0.793172] EDAC MC: Ver: 3.0.0 [ 0.796172] PCI: Using ACPI for IRQ routing [ 0.797900] NetLabel: Initializing [ 0.799019] NetLabel: domain hash size = 128 [ 0.801015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.803088] NetLabel: unlabeled traffic allowed by default [ 0.806153] vgaarb: loaded [ 0.808189] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.810020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.819017] clocksource: Switched to clocksource kvm-clock [ 0.927433] VFS: Disk quotas dquot_6.6.0 [ 0.929563] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.932043] *** VALIDATE ramfs *** [ 0.933289] *** VALIDATE hugetlbfs *** [ 0.935110] pnp: PnP ACPI init [ 0.937185] pnp: PnP ACPI: found 6 devices [ 0.964659] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.968458] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.970963] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.973197] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.975507] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.978167] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.980802] NET: Registered protocol family 2 [ 0.983431] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.987159] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.990857] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.996330] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.999911] TCP: Hash tables configured (established 65536 bind 65536) [ 1.003098] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.006608] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.009537] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.012520] NET: Registered protocol family 1 [ 1.015747] RPC: Registered named UNIX socket transport module. [ 1.018095] RPC: Registered udp transport module. [ 1.020246] RPC: Registered tcp transport module. [ 1.022146] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.024389] NET: Registered protocol family 44 [ 1.026317] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.029166] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.031947] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.035033] PCI: CLS 0 bytes, default 64 [ 1.036932] Unpacking initramfs... [ 2.521722] debug: unmapping init [mem 0xffff92c17cc54000-0xffff92c17ffbffff] [ 2.529062] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.531027] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.533531] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.022928] Initialise system trusted keyrings [ 3.024836] Key type blacklist registered [ 3.027076] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.038050] zbud: loaded [ 3.041644] *** VALIDATE nfs *** [ 3.042642] *** VALIDATE nfs4 *** [ 3.043851] pstore: using deflate compression [ 3.048231] Platform Keyring initialized [ 3.158498] NET: Registered protocol family 38 [ 3.160078] Key type asymmetric registered [ 3.161637] Asymmetric key parser 'x509' registered [ 3.163732] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.167109] io scheduler mq-deadline registered [ 3.169116] io scheduler kyber registered [ 3.170921] io scheduler bfq registered [ 3.172929] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.176166] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.179046] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.181526] ACPI: Power Button [PWRF] [ 3.187354] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.194438] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.211111] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.219929] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.233961] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.261642] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.289269] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.294468] Non-volatile memory driver v1.3 [ 3.296308] Linux agpgart interface v0.103 [ 3.326815] virtio_blk virtio1: [vda] 149816 512-byte logical blocks (76.7 MB/73.2 MiB) [ 3.330148] vda: detected capacity change from 0 to 76705792 [ 3.347112] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.349924] vdb: detected capacity change from 0 to 1073741824 [ 3.368482] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.372303] vdc: detected capacity change from 0 to 2621440000 [ 3.389475] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.391904] vdd: detected capacity change from 0 to 2621440000 [ 3.410356] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.413573] vde: detected capacity change from 0 to 4294967296 [ 3.432405] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.435729] vdf: detected capacity change from 0 to 4294967296 [ 3.442422] libphy: Fixed MDIO Bus: probed [ 3.447891] usbcore: registered new interface driver usbserial_generic [ 3.450975] usbserial: USB Serial support registered for generic [ 3.453414] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.457598] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.459488] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.461701] mousedev: PS/2 mouse device common for all mice [ 3.464743] rtc_cmos 00:05: RTC can wake from S4 [ 3.467172] rtc_cmos 00:05: registered as rtc0 [ 3.468890] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.469661] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.475812] intel_pstate: CPU model not supported [ 3.477859] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.483273] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.483653] hid: raw HID events driver (C) Jiri Kosina [ 3.488121] usbcore: registered new interface driver usbhid [ 3.490395] usbhid: USB HID core driver [ 3.492124] drop_monitor: Initializing network drop monitor service [ 3.494688] Initializing XFRM netlink socket [ 3.496798] NET: Registered protocol family 10 [ 3.499815] Segment Routing with IPv6 [ 3.501380] NET: Registered protocol family 17 [ 3.503744] mpls_gso: MPLS GSO support [ 3.510177] RAS: Correctable Errors collector initialized. [ 3.512352] AVX version of gcm_enc/dec engaged. [ 3.513558] AES CTR mode by8 optimization enabled [ 3.593711] sched_clock: Marking stable (3593687941, 0)->(4545554470, -951866529) [ 3.597777] registered taskstats version 1 [ 3.600471] Loading compiled-in X.509 certificates [ 3.603015] zswap: loaded using pool lzo/zbud [ 3.631500] Key type big_key registered [ 3.643885] Key type encrypted registered [ 3.645546] ima: No TPM chip found, activating TPM-bypass! [ 3.648784] ima: Allocated hash algorithm: sha1 [ 3.650864] ima: No architecture policies found [ 3.652640] evm: Initialising EVM extended attributes: [ 3.654920] evm: security.selinux [ 3.656446] evm: security.ima [ 3.657799] evm: security.capability [ 3.659157] evm: HMAC attrs: 0x1 [ 3.661708] rtc_cmos 00:05: setting system clock to 2026-09-05 07:32:57 UTC (1788593577) [ 3.668394] debug: unmapping init [mem 0xffffffff85803000-0xffffffff859fffff] [ 3.672519] debug: unmapping init [mem 0xffffffff84582000-0xffffffff84858fff] [ 3.685131] Write protecting the kernel read-only data: 28672k [ 3.688730] debug: unmapping init [mem 0xffffffff82c03000-0xffffffff82dfffff] [ 3.691945] debug: unmapping init [mem 0xffffffff83514000-0xffffffff835fffff] [ 3.735591] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.745290] systemd[1]: Detected virtualization kvm. [ 3.747255] systemd[1]: Detected architecture x86-64. [ 3.749221] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.775505] systemd[1]: No hostname configured. [ 3.777669] systemd[1]: Set hostname to . [ 3.779920] random: systemd: uninitialized urandom read (16 bytes read) [ 3.782406] systemd[1]: Initializing machine ID from random generator. [ 3.834275] random: ln: uninitialized urandom read (6 bytes read) [ 3.927570] random: systemd: uninitialized urandom read (16 bytes read) [ 3.930914] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.936041] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.940885] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.624239] device-mapper: uevent: version 1.0.3 [ 4.627673] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.397418] random: fast init done [ 5.412128] virtio_net virtio0 ens2: renamed from eth0 [ 5.453559] scsi host0: ata_piix [ 5.472762] scsi host1: ata_piix [ 5.486582] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.489242] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.852345] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.163985] random: crng init done [ 10.165312] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.643514] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.867115] printk: systemd: 19 output lines suppressed due to ratelimiting [ 12.142144] SELinux: Disabled at runtime. [ 12.206681] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.215779] systemd[1]: Detected virtualization kvm. [ 12.218070] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.761974] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.768710] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.772915] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.778436] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.782240] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.790108] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.795108] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Reached targ[ 12.916550] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS et Slices. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.262286] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.593445] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.651895] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.846055] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.865528] EDAC sbridge: Ver: 1.1.2 [ 15.573850] Key type dns_resolver registered [ 15.890857] NFS: Registering the id_resolver key type [ 15.893361] Key type id_resolver registered [ 15.894983] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg445-server login: [ 73.737123] hrtimer: interrupt took 5109705 ns [ 83.558601] libcfs: loading out-of-tree module taints kernel. [ 83.675431] Key type ._llcrypt registered [ 83.678282] Key type .llcrypt registered [ 83.838794] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 103.443481] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 104.778226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 104.794875] alg: No test for adler32 (adler32-zlib) [ 106.457142] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 107.470450] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 109.360314] Key type lgssc registered [ 110.947389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 129.392270] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.717560] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 186.230959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 186.274747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.503797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 187.529396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 187.614948] Lustre: lustre-MDT0000: new disk, initializing [ 187.730910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.764274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 192.472562] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.147207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.225719] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 206.296699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 206.304082] Lustre: Skipped 1 previous similar message [ 206.435612] Lustre: lustre-MDT0001: new disk, initializing [ 206.531801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 206.565461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 206.581223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 211.256874] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.177618] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 226.879771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 227.361361] Lustre: lustre-OST0000: new disk, initializing [ 227.365088] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 227.377526] Lustre: 8453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 227.448964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 232.455594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 232.462540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 232.501643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 233.549594] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.018543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.205923] Lustre: lustre-OST0001: new disk, initializing [ 248.209591] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 248.215711] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.284809] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 254.785612] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.015226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 256.025091] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 256.062610] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 266.460543] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 271.146907] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 277.301369] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing check_logdir /tmp/testlogs/ [ 282.997757] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing yml_node [ 287.238845] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 289.696152] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 292.410317] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 294.061725] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 03:37:46 EDT 2026 [ 311.039392] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 312.671901] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 329.697316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 329.703761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.716166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.776137] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.777206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.798670] Lustre: Skipped 2 previous similar messages [ 332.802268] Lustre: Skipped 2 previous similar messages [ 334.639325] Lustre: server umount lustre-MDT0000 complete [ 338.176366] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593912 with bad export cookie 2939269895328707132 [ 338.181409] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.187960] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 343.015307] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.015592] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.035440] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 343.040314] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.051525] Lustre: Skipped 1 previous similar message [ 343.057805] Lustre: Skipped 1 previous similar message [ 344.565538] Lustre: server umount lustre-MDT0001 complete [ 348.416681] Lustre: server umount lustre-OST0000 complete [ 352.234365] Lustre: server umount lustre-OST0001 complete [ 359.237505] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 369.402332] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 404.001416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 404.200038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 404.231995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 404.292254] Lustre: lustre-MDT0000: new disk, initializing [ 404.340148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 404.353871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 408.005322] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 420.673466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.857873] Lustre: 15084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 420.932891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 420.936460] Lustre: Skipped 1 previous similar message [ 421.027847] Lustre: lustre-MDT0001: new disk, initializing [ 421.178272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 421.235891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 421.254828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 426.672359] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 437.750592] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 444.626657] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 450.852977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 451.108507] Lustre: lustre-OST0000: new disk, initializing [ 451.113256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 451.119588] Lustre: 17054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 451.249248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.352498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 452.369684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 452.461481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 456.793715] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 465.391251] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 469.605128] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 469.764054] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 473.658952] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 473.855548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 478.189878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 478.193216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 478.195050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 478.195060] Lustre: Skipped 1 previous similar message [ 478.201025] Lustre: Skipped 1 previous similar message [ 478.226185] LustreError: Skipped 1 previous similar message [ 482.231813] Lustre: server umount lustre-OST0000 complete [ 487.905672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 487.911167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.922662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 487.926607] Lustre: Skipped 1 previous similar message [ 492.472104] Lustre: server umount lustre-MDT0000 complete [ 493.539458] LustreError: 15095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.562336] LustreError: 15095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 496.396312] LustreError: 17052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594070 with bad export cookie 2939269895328708665 [ 496.400198] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 496.409046] LustreError: 17052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 496.709113] Lustre: server umount lustre-MDT0001 complete [ 500.508797] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 03:41:12 (1788594072) [ 502.861103] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 504.597854] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 03:41:16 (1788594076) [ 510.766960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.206260] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.280415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 516.462613] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 516.580241] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.706779] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.692413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.978862] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 536.489881] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 542.135804] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 548.822275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 555.739877] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 563.374424] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 571.873354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 571.891970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.895139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 571.913131] Lustre: Skipped 2 previous similar messages [ 571.924503] Lustre: Skipped 3 previous similar messages [ 577.532237] Lustre: server umount lustre-OST0000 complete [ 588.023476] Lustre: server umount lustre-MDT0000 complete [ 591.229143] LustreError: 19443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594165 with bad export cookie 2939269895328709547 [ 591.239205] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.245507] LustreError: 19443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 591.483697] Lustre: server umount lustre-MDT0001 complete [ 605.167225] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 608.505285] Key type lgssc unregistered [ 608.879354] LNet: 22995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.891814] LNetError: 22995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 608.937047] LNet: Removed LNI 192.168.204.145@tcp [ 609.971157] Key type .llcrypt unregistered [ 609.983547] Key type ._llcrypt unregistered [ 634.980052] Key type ._llcrypt registered [ 634.981927] Key type .llcrypt registered [ 635.089452] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 636.431772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 636.465659] alg: No test for adler32 (adler32-zlib) [ 637.590405] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 637.858239] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 639.520221] Key type lgssc registered [ 640.529113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 648.736899] loop: module loaded [ 648.820183] loop0: detected capacity change from 0 to 204800000 [ 649.058176] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 649.074667] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 649.536152] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.042938] loop0: detected capacity change from 0 to 204800000 [ 656.171458] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.195210] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.209845] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 661.752868] loop0: detected capacity change from 0 to 204800000 [ 668.043907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 668.082122] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.430792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 669.515095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 669.603434] Lustre: lustre-MDT0000: new disk, initializing [ 669.714237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.738704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 674.103783] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 684.333307] loop1: detected capacity change from 0 to 204800000 [ 691.121964] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 691.433921] Lustre: lustre-OST07c6: new disk, initializing [ 691.437612] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 691.443629] Lustre: Skipped 1 previous similar message [ 691.448306] Lustre: 26179:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 691.512807] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 697.694748] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 698.527545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 698.539552] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 698.581597] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 703.927263] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 703.934815] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 703.948136] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 703.953686] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 703.956800] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 703.960593] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 717.343628] Lustre: 26559:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 717.355872] Lustre: 26559:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 717.360779] Lustre: 26559:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 717.367504] Lustre: 26559:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 717.375625] Lustre: 26559:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 717.381604] Lustre: 26559:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 723.128588] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 723.138178] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 723.155333] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 723.167410] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.180985] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 723.191777] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.199790] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 723.209360] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.236835] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 723.251746] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.273412] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 723.283839] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.543749] Lustre: 26559:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 726.557362] Lustre: 26559:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 726.565177] Lustre: 26559:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 726.573468] Lustre: 26559:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.581224] Lustre: 26559:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 726.595601] Lustre: 26559:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.604633] Lustre: 26559:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 726.611777] Lustre: 26559:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.621499] Lustre: 26559:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 726.633781] Lustre: 26559:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.649307] Lustre: 26559:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 726.661361] Lustre: 26559:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 735.094961] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 735.106140] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 735.118604] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 735.125309] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 735.131773] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 735.140888] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 735.147117] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 735.153090] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 735.156570] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 735.161425] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 735.168833] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 735.174144] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 746.268942] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 746.282770] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 746.290688] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 746.301738] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 746.312903] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 746.321704] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 746.327674] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 746.332955] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 746.341732] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 746.355835] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 746.364649] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 746.373509] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 790.499226] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 790.516541] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 792.687355] Lustre: server umount lustre-OST07c6 complete [ 803.030717] Lustre: server umount lustre-MDT0000 complete [ 821.737497] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 824.496295] Key type lgssc unregistered [ 824.765515] LNet: 28734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.771783] LNetError: 28734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.796317] LNet: Removed LNI 192.168.204.145@tcp [ 825.745338] Key type .llcrypt unregistered [ 825.753272] Key type ._llcrypt unregistered [ 846.609415] Key type ._llcrypt registered [ 846.617634] Key type .llcrypt registered [ 846.795553] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 848.432766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 848.508806] alg: No test for adler32 (adler32-zlib) [ 849.607639] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 850.075120] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 851.808200] Key type lgssc registered [ 853.006931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 860.831667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 860.849261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 862.471810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 867.453831] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 872.933913] LustreError: 30000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 872.950401] LustreError: 30000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 877.116496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.557264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 882.100474] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 890.571685] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 896.046961] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 901.923764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 902.246193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.384458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 909.077849] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 916.671457] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 933.348409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 933.357067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.362564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.386937] Lustre: Skipped 1 previous similar message [ 933.414575] Lustre: Skipped 1 previous similar message [ 933.885462] Lustre: server umount lustre-OST0000 complete [ 938.475519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 938.485420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 938.494614] Lustre: Skipped 1 previous similar message [ 938.514073] Lustre: Skipped 1 previous similar message [ 943.588316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.601372] Lustre: Skipped 1 previous similar message [ 944.576767] Lustre: server umount lustre-MDT0000 complete [ 948.706852] LustreError: 30001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.738417] LustreError: 30001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 949.111619] LustreError: 29981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594522 with bad export cookie 16724702382960995560 [ 949.119671] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 949.124691] LustreError: 29981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 949.548920] Lustre: server umount lustre-MDT0001 complete [ 962.525373] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 966.089895] Key type lgssc unregistered [ 966.512580] LNet: 33738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.526916] LNetError: 33738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.592527] LNet: Removed LNI 192.168.204.145@tcp [ 968.787295] Key type .llcrypt unregistered [ 968.790178] Key type ._llcrypt unregistered [ 990.623923] Key type ._llcrypt registered [ 990.629518] Key type .llcrypt registered [ 990.808485] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 992.386955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 992.435275] alg: No test for adler32 (adler32-zlib) [ 993.746379] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 994.179506] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 996.016167] Key type lgssc registered [ 998.050226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1008.852805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1008.874259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1010.543850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1015.688641] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1020.915260] LustreError: 35003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.962128] LustreError: 35003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1026.033330] LustreError: 35004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.183226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.508532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1032.339636] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1042.466219] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1049.556231] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1057.563793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1058.038890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1066.193408] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1074.886492] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1081.607465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1092.069305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1092.072489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.077091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1092.086995] LustreError: Skipped 1 previous similar message [ 1092.094353] Lustre: Skipped 1 previous similar message [ 1096.633362] Lustre: server umount lustre-OST0000 complete [ 1102.310716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1102.313886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1102.336124] Lustre: Skipped 1 previous similar message [ 1102.348539] Lustre: Skipped 2 previous similar messages [ 1107.425810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1107.436200] Lustre: Skipped 1 previous similar message [ 1108.214650] Lustre: server umount lustre-MDT0000 complete [ 1112.546922] LustreError: 34996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1112.556136] LustreError: 34996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1113.476768] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594687 with bad export cookie 12746256536874587490 [ 1113.478129] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1113.492053] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1114.195486] Lustre: server umount lustre-MDT0001 complete [ 1129.279675] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1132.958280] Key type lgssc unregistered [ 1133.445093] LNet: 38561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.453777] LNetError: 38561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.487358] LNet: Removed LNI 192.168.204.145@tcp [ 1134.606175] Key type .llcrypt unregistered [ 1134.608658] Key type ._llcrypt unregistered [ 1163.067301] Key type ._llcrypt registered [ 1163.072048] Key type .llcrypt registered [ 1163.310889] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1165.125722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1165.168917] alg: No test for adler32 (adler32-zlib) [ 1166.436436] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 1166.965274] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1168.704186] Key type lgssc registered [ 1170.030099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1179.081733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1179.117435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.639411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1186.355644] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1190.887345] LustreError: 39976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.901908] LustreError: 39976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1196.016277] LustreError: 39975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.562276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1197.859189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1204.429514] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1214.429386] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1222.564332] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1231.426363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1231.998381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.656597] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1250.439792] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1250.443637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1258.836448] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1259.493450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1259.502987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.518109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1261.031421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1261.034133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.047053] Lustre: Skipped 1 previous similar message [ 1261.069680] Lustre: Skipped 1 previous similar message [ 1263.903979] LustreError: 39970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.080760] Lustre: server umount lustre-MDT0000 complete [ 1266.161498] LustreError: 39976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.195458] LustreError: 39976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1268.296633] LustreError: 39954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594842 with bad export cookie 13836486028141047586 [ 1268.310524] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1268.316274] LustreError: 39954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1269.049261] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [ 1269.062828] Lustre: Skipped 1 previous similar message [ 1271.265991] LustreError: 42455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1271.270826] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.283438] LustreError: 42455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1273.316271] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1273.326276] Lustre: Skipped 1 previous similar message [ 1274.784810] Lustre: server umount lustre-MDT0001 complete [ 1296.497744] Lustre: server umount lustre-OST0000 complete [ 1315.906187] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1319.160462] Key type lgssc unregistered [ 1319.474591] LNet: 43844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1319.489916] LNetError: 43844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1319.511077] LNet: Removed LNI 192.168.204.145@tcp [ 1320.507490] Key type .llcrypt unregistered [ 1320.510403] Key type ._llcrypt unregistered [ 1348.414117] Key type ._llcrypt registered [ 1348.416653] Key type .llcrypt registered [ 1348.591254] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1349.939936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1349.986070] alg: No test for adler32 (adler32-zlib) [ 1351.435806] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 1351.811959] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1353.464183] Key type lgssc registered [ 1354.573780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1364.464704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1364.486114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1366.094184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.628186] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1375.207837] LustreError: 45257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.244491] LustreError: 45257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1380.336052] LustreError: 45258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1382.303790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1382.788735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1388.283671] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1397.151540] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1403.793237] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1411.627206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1412.067575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1419.735519] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1427.568336] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1435.586478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1445.858348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1445.887674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.894733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.909397] Lustre: Skipped 1 previous similar message [ 1445.927077] Lustre: Skipped 1 previous similar message [ 1449.795647] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1450.990259] LustreError: 47081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1451.008122] LustreError: 47081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1451.169800] Lustre: server umount lustre-OST0000 complete [ 1470.581186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1470.596395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1470.611204] Lustre: Skipped 1 previous similar message [ 1470.628075] Lustre: Skipped 1 previous similar message [ 1475.750184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1475.763577] Lustre: Skipped 1 previous similar message [ 1477.195841] Lustre: server umount lustre-MDT0000 complete [ 1481.199594] LustreError: 45258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1481.216498] LustreError: 45258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1481.372419] LustreError: 45238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595055 with bad export cookie 2209093036039833774 [ 1481.379754] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1481.386360] LustreError: 45238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1487.809341] Lustre: server umount lustre-MDT0001 complete [ 1501.128655] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1504.376119] Key type lgssc unregistered [ 1504.679885] LNet: 48989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1504.696351] LNetError: 48989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1504.714773] LNet: Removed LNI 192.168.204.145@tcp [ 1505.708101] Key type .llcrypt unregistered [ 1505.710516] Key type ._llcrypt unregistered [ 1530.807676] Key type ._llcrypt registered [ 1530.809659] Key type .llcrypt registered [ 1530.912296] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1531.792365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1531.835615] alg: No test for adler32 (adler32-zlib) [ 1532.908431] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 1533.133189] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1534.824225] Key type lgssc registered [ 1535.782981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1544.338828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1544.360107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1545.731825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.733381] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1555.938114] LustreError: 50402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1555.968449] LustreError: 50402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1561.058364] LustreError: 50403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1562.404098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.858210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1566.875435] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1575.480079] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1581.852675] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1589.393318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1589.891894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1590.923156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1596.997853] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1604.088465] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1612.509779] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1615.823163] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1627.763479] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-05 3h59m59s [ 1629.141766] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1631.546985] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1632.593851] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1633.258648] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1633.278650] Lustre: Skipped 1 previous similar message [ 1634.345520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1634.350027] Lustre: Skipped 5 previous similar messages [ 1637.379220] Lustre: server umount lustre-MDT0000 complete [ 1637.641712] LustreError: 52895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.419616] LustreError: 50402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.444933] LustreError: 50402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1641.070223] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1646.563661] LustreError: 51130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1646.590754] LustreError: 51130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1648.269332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1648.371891] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1648.557038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1648.641528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1649.588191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1654.644354] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1659.255786] Lustre: DEBUG MARKER: Wait for df (36677) ... [ 1660.835775] Lustre: DEBUG MARKER: done [ 1669.091959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1669.094157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.100043] LustreError: Skipped 1 previous similar message [ 1669.101156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1669.117035] Lustre: Skipped 2 previous similar messages [ 1671.631404] Lustre: server umount lustre-OST0000 complete [ 1675.234428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1675.241359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1675.250328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1675.257627] Lustre: Skipped 1 previous similar message [ 1675.263335] Lustre: Skipped 3 previous similar messages [ 1681.236507] Lustre: server umount lustre-MDT0000 complete [ 1681.381323] LustreError: 51130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1681.399811] LustreError: 51130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1685.400547] LustreError: 50381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595259 with bad export cookie 8580114762051524349 [ 1685.409552] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1685.412428] LustreError: 50381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1685.995823] Lustre: server umount lustre-MDT0001 complete [ 1699.825701] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1702.971898] Key type lgssc unregistered [ 1703.402638] LNet: 55341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.410157] LNetError: 55341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.433060] LNet: Removed LNI 192.168.204.145@tcp [ 1704.293262] Key type .llcrypt unregistered [ 1704.295924] Key type ._llcrypt unregistered [ 1744.740491] Key type ._llcrypt registered [ 1744.744169] Key type .llcrypt registered [ 1744.851896] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1746.654673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1746.719108] alg: No test for adler32 (adler32-zlib) [ 1747.987795] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 1748.344671] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1750.120223] Key type lgssc registered [ 1751.544371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1760.258211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1760.272675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1761.498631] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1761.516777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1761.669354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.871273] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1774.869277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1774.949670] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1774.987958] Lustre: 57544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1775.285776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1779.564921] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1786.532271] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1792.186304] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1799.384395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1799.610979] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1799.798646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1805.971288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1806.619228] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1814.148450] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1821.596691] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-05 4h03m13s [ 1823.275333] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1830.987156] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1833.629163] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1833.695842] LustreError: 60072:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1853.738584] Lustre: lustre-MDT0000: Client 271d7d44-fb9e-42ac-bebc-cc10ca85b4a4 (at 192.168.204.45@tcp) reconnecting [ 1873.720844] LustreError: 60072:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1876.144900] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1878.738779] Lustre: DEBUG MARKER: done [ 1886.580829] Lustre: server umount lustre-OST0000 complete [ 1893.347317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1893.349113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1893.364670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1893.383475] Lustre: Skipped 1 previous similar message [ 1893.389935] Lustre: Skipped 1 previous similar message [ 1894.437508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1894.444027] Lustre: Skipped 1 previous similar message [ 1900.522746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1900.539575] Lustre: Skipped 1 previous similar message [ 1900.659766] Lustre: server umount lustre-MDT0000 complete [ 1903.589177] Lustre: 56866:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788595461/real 1788595461] req@0000000097cf5569 x1875478128416640/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788595477 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1903.626912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1904.257937] LustreError: 58286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595478 with bad export cookie 16521169279623613554 [ 1904.263818] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1904.275215] LustreError: 58286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1904.565502] Lustre: server umount lustre-MDT0001 complete [ 1914.082046] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1917.207513] Key type lgssc unregistered [ 1917.630772] LNet: 61870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.645291] LNetError: 61870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1918.699838] LNet: Removed LNI 192.168.204.145@tcp [ 1920.178442] Key type .llcrypt unregistered [ 1920.182220] Key type ._llcrypt unregistered [ 1965.817611] Key type ._llcrypt registered [ 1965.820967] Key type .llcrypt registered [ 1965.909606] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1967.207524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1967.306293] alg: No test for adler32 (adler32-zlib) [ 1968.303983] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 1968.637458] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1970.280314] Key type lgssc registered [ 1971.218788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1974.763594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1974.778959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.206907] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1976.230583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1976.410839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1977.366736] Lustre: Failing over lustre-MDT0000 [ 1977.584403] Lustre: server umount lustre-MDT0000 complete [ 1983.310592] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 04:05:55 (1788595555) [ 1990.065939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1990.604613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.901869] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2002.042994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.118318] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2002.140181] Lustre: 64668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2002.359673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2006.781676] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2015.009590] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2020.400651] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2027.936246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2028.240896] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2028.609487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2034.568308] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2036.362437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2042.173464] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2049.160750] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2055.163107] LustreError: 63591:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a1e6e5bc x1875478359084288/t0(0) o2->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2057.191158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2057.199501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2057.212990] Lustre: Skipped 1 previous similar message [ 2061.598910] Lustre: server umount lustre-MDT0000 complete [ 2065.771433] LustreError: 64661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595639 with bad export cookie 12281040688746297880 [ 2065.775650] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2065.795131] LustreError: 64661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2066.296456] Lustre: server umount lustre-MDT0001 complete [ 2069.967962] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2074.619588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2083.190080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2083.809836] Lustre: 63589:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788595641/real 1788595641] req@0000000080a722ba x1875478359089024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788595657 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2083.832298] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.841566] Lustre: Skipped 1 previous similar message [ 2086.944167] Lustre: 63589:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788595644/real 1788595644] req@000000007fa7f791 x1875478359089152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788595660 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2088.928203] Lustre: 63591:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788595646/real 1788595646] req@0000000066f2950f x1875478359089280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788595662 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2091.044029] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xaa6f05704cd782fb [ 2091.060996] Lustre: MGC192.168.204.145@tcp: Connection restored to 0@lo (at 0@lo) [ 2091.359798] LustreError: 66515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2091.381800] LustreError: 66515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2091.463229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.578692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2092.449117] Lustre: 63592:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788595649/real 1788595649] req@00000000728bfd90 x1875478359089536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788595665 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2092.474136] LustreError: 66516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.543187] LustreError: 68439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.573923] LustreError: 68439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2096.292548] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2096.616422] Lustre: 63589:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788595654/real 1788595654] req@000000002f5bffd5 x1875478359089664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788595670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2098.659091] LustreError: 68441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2098.699885] LustreError: 68441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2103.785067] LustreError: 66515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.824100] LustreError: 66515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2105.564164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2105.923416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2110.255362] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2110.957381] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2112.998221] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2117.244823] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2123.075272] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2131.426613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2131.432829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.453312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2131.458278] Lustre: Skipped 2 previous similar messages [ 2133.477845] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.481091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2133.492343] Lustre: Skipped 1 previous similar message [ 2133.500432] Lustre: Skipped 2 previous similar messages [ 2135.633834] Lustre: server umount lustre-MDT0000 complete [ 2138.594443] LustreError: 68440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2138.625718] LustreError: 68440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2139.082289] LustreError: 64660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595712 with bad export cookie 12281040688746300155 [ 2139.087553] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2139.097098] LustreError: 64660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2139.346029] Lustre: server umount lustre-MDT0001 complete [ 2145.028353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2154.472983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2160.096214] Lustre: 63592:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788595717/real 1788595717] req@00000000fc6cff2c x1875478359117696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788595733 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2160.131157] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2163.571840] LustreError: 69176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.666281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.734385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2168.509113] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2178.526353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2184.171355] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2185.082529] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2188.275754] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2194.541900] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2201.233794] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2208.740192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2208.741607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2208.754089] Lustre: Skipped 1 previous similar message [ 2208.766287] Lustre: Skipped 2 previous similar messages [ 2213.860172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2213.880834] Lustre: Skipped 2 previous similar messages [ 2214.716140] Lustre: server umount lustre-MDT0000 complete [ 2218.755284] LustreError: 64661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595792 with bad export cookie 12281040688746301534 [ 2218.763065] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2218.778690] LustreError: 64661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2219.011520] Lustre: server umount lustre-MDT0001 complete [ 2226.143199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2231.418638] Lustre: DEBUG MARKER: files compared the same [ 2243.826045] Lustre: server umount lustre-OST0000 complete [ 2262.739727] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2265.409759] Key type lgssc unregistered [ 2265.701367] LNet: 74398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2265.711720] LNetError: 74398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2265.740564] LNet: Removed LNI 192.168.204.145@tcp [ 2266.616882] Key type .llcrypt unregistered [ 2266.620102] Key type ._llcrypt unregistered [ 2290.790406] Key type ._llcrypt registered [ 2290.793103] Key type .llcrypt registered [ 2290.905799] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2292.055118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2292.093756] alg: No test for adler32 (adler32-zlib) [ 2293.071669] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 2293.256664] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2295.000748] Key type lgssc registered [ 2296.057179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2302.982280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2303.027124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2304.546210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.544900] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug +malloc all [ 2314.728887] LustreError: 75813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.745517] LustreError: 75813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2318.315715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.678193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2323.854157] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug +malloc all [ 2331.731605] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2337.330556] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2344.036890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2344.325470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.489794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2350.834465] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug +malloc all [ 2359.215191] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2370.019305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2370.029231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.046634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2374.115905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2374.118245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2374.121166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.137929] Lustre: Skipped 1 previous similar message [ 2379.239951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2384.353244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2384.356137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 2384.360068] Lustre: Skipped 2 previous similar messages [ 2384.521802] Lustre: server umount lustre-OST0000 complete [ 2389.472918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2389.477363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2389.496694] Lustre: Skipped 1 previous similar message [ 2389.508283] Lustre: Skipped 1 previous similar message [ 2399.712922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2399.719120] Lustre: Skipped 3 previous similar messages [ 2403.296131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2403.413157] Lustre: server umount lustre-MDT0000 complete [ 2404.833930] LustreError: 75807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.862083] LustreError: 75807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2407.256232] LustreError: 75793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595981 with bad export cookie 7148050863608690421 [ 2407.266765] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2407.273300] LustreError: 75793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2407.573794] Lustre: server umount lustre-MDT0001 complete [ 2416.877346] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2419.658515] Key type lgssc unregistered [ 2419.988977] LNet: 79346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2420.009463] LNetError: 79346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.035881] LNet: Removed LNI 192.168.204.145@tcp [ 2420.793870] Key type .llcrypt unregistered [ 2420.799349] Key type ._llcrypt unregistered [ 2444.436815] Key type ._llcrypt registered [ 2444.439212] Key type .llcrypt registered [ 2444.540814] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2446.157563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2446.195148] alg: No test for adler32 (adler32-zlib) [ 2447.534721] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 2447.803340] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2449.496753] Key type lgssc registered [ 2450.770924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2458.679628] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2458.692932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2474.272780] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2479.396223] LustreError: 80734:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 2506.080743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2512.459894] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2520.066521] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2527.074610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.230861] Lustre: *** cfs_fail_loc=706, val=0*** [ 2527.239771] LustreError: 80068:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000056050854 x1875478861318400/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2527.527510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.612625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2532.195871] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2532.838902] LustreError: 80760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2532.862281] LustreError: 80760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2537.956935] LustreError: 80759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.245949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.569050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2545.056737] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2552.416617] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2557.950677] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2560.998651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2561.020953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2561.033541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2562.022077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2562.026484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2566.677460] Lustre: server umount lustre-OST0000 complete [ 2571.235077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2571.256526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.289020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2571.301116] Lustre: Skipped 1 previous similar message [ 2577.380831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2577.388948] Lustre: Skipped 3 previous similar messages [ 2582.498862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2582.512329] Lustre: Skipped 1 previous similar message [ 2585.056201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2585.223800] Lustre: server umount lustre-MDT0000 complete [ 2586.593876] LustreError: 81829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.616838] LustreError: 81829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2588.984410] LustreError: 80742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596162 with bad export cookie 5927528070415829850 [ 2588.999532] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2589.341982] Lustre: server umount lustre-MDT0001 complete [ 2601.525850] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2604.737202] Key type lgssc unregistered [ 2605.042937] LNet: 84354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2605.052161] LNetError: 84354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2605.069725] LNet: Removed LNI 192.168.204.145@tcp [ 2605.948307] Key type .llcrypt unregistered [ 2605.950457] Key type ._llcrypt unregistered [ 2631.266531] Key type ._llcrypt registered [ 2631.268178] Key type .llcrypt registered [ 2631.402714] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2632.741699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2632.773476] alg: No test for adler32 (adler32-zlib) [ 2633.895832] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 2634.160243] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2635.898410] Key type lgssc registered [ 2637.373630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2646.005265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2646.029985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.366100] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2658.162211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.727471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2662.913443] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2663.906414] LustreError: 86454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2663.931078] LustreError: 86454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2669.027515] LustreError: 86458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.046704] LustreError: 86458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2674.159301] LustreError: 86457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.189511] LustreError: 86457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2674.215634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2674.544302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2679.777141] LustreError: 86458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.793547] LustreError: 86458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2682.220739] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2683.880771] LustreError: 86457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.906185] LustreError: 86457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2690.666604] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2690.680566] Lustre: MGS: Client d3763289-5501-4c39-9b76-0bc593b0dfec (at 0@lo) reconnecting [ 2690.735688] LustreError: 86468:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2690.991411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2691.104644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2695.190890] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2711.521627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2711.523245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.527021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2711.530939] Lustre: Skipped 1 previous similar message [ 2714.391990] Lustre: server umount lustre-OST0000 complete [ 2721.763634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2721.766296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.775690] LustreError: Skipped 1 previous similar message [ 2721.780630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2721.780637] Lustre: Skipped 1 previous similar message [ 2721.821791] Lustre: Skipped 1 previous similar message [ 2723.775295] Lustre: server umount lustre-MDT0000 complete [ 2726.882162] LustreError: 86457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.911105] LustreError: 86457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2727.852589] Lustre: server umount lustre-MDT0001 complete [ 2732.135511] Lustre: server umount MGS complete [ 2746.224642] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:18:37 (1788596317) [ 2776.943973] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 2788.152726] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2827.500340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.746295] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2842.463614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.721460] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2842.793167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2842.932267] Lustre: lustre-MDT0001: new disk, initializing [ 2843.052208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2843.106732] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2843.113654] Lustre: 93625:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2843.125109] LustreError: 93625:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2848.324366] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2859.643384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2859.943532] Lustre: lustre-OST0000: new disk, initializing [ 2859.952168] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2859.974164] Lustre: 94391:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2860.054353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.073601] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2869.349690] Lustre: 94847:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ef823f5e x1875479056392448/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788596459 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2875.646781] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2882.984576] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2882.997118] Lustre: MGS: Client 84c073c2-df73-477a-83db-ac07eb03ff34 (at 0@lo) reconnecting [ 2883.074773] Lustre: 92891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2883.093939] LustreError: 93611:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2883.116641] LustreError: 93611:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2883.162346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2883.281866] Lustre: lustre-MDT0000: new disk, initializing [ 2883.439278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2883.464751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2883.562878] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2884.528103] Lustre: 95460:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000604b0ad3 x1875479056399104/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788596474 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2888.552509] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2889.764782] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2893.616130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2893.639282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2893.868241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2919.912212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2919.923072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.939044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2919.952825] Lustre: Skipped 1 previous similar message [ 2925.649232] Lustre: server umount lustre-OST0000 complete [ 2931.169749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2931.179986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2931.182916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2931.182924] Lustre: Skipped 2 previous similar messages [ 2931.215544] Lustre: Skipped 2 previous similar messages [ 2936.294418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2936.306414] Lustre: Skipped 2 previous similar messages [ 2936.424721] Lustre: server umount lustre-MDT0000 complete [ 2940.397214] LustreError: 93594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.429762] LustreError: 93594:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2946.840201] Lustre: server umount lustre-MDT0001 complete [ 2950.358489] Lustre: server umount MGS complete [ 2961.456828] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 04:22:13 (1788596533) [ 2980.541722] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2983.938897] Key type lgssc unregistered [ 2984.276233] LNet: 97958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2984.287558] LNetError: 97958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2984.316822] LNet: Removed LNI 192.168.204.145@tcp [ 2985.336478] Key type .llcrypt unregistered [ 2985.342341] Key type ._llcrypt unregistered [ 3006.193536] Key type ._llcrypt registered [ 3006.195722] Key type .llcrypt registered [ 3006.331231] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3008.036532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3008.081101] alg: No test for adler32 (adler32-zlib) [ 3009.182843] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 3009.467415] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3011.192455] Key type lgssc registered [ 3012.471897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3021.542870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3021.570076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3023.090732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3028.642267] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3032.550554] LustreError: 99181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.585792] LustreError: 99181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3037.675562] LustreError: 99182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.489459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.051032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3044.592156] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3052.954550] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3059.937546] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3068.048475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3068.478794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.266908] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3083.048378] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3093.050549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3103.202824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3103.204184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.249500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3108.224855] Lustre: server umount lustre-OST0000 complete [ 3113.443788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3113.456077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3113.475588] Lustre: Skipped 2 previous similar messages [ 3113.480172] Lustre: Skipped 1 previous similar message [ 3118.562296] LustreError: 99181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.570991] Lustre: server umount lustre-MDT0000 complete [ 3122.746154] LustreError: 99162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596696 with bad export cookie 1333085481189326434 [ 3122.748223] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3122.763016] LustreError: 99162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3123.156937] Lustre: server umount lustre-MDT0001 complete [ 3134.448412] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3137.176332] Key type lgssc unregistered [ 3137.412107] LNet: 102734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3137.417518] LNetError: 102734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3138.473970] LNet: Removed LNI 192.168.204.145@tcp [ 3139.343915] Key type .llcrypt unregistered [ 3139.347075] Key type ._llcrypt unregistered [ 3157.254050] Key type ._llcrypt registered [ 3157.276694] Key type .llcrypt registered [ 3157.521736] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3158.877207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3158.917972] alg: No test for adler32 (adler32-zlib) [ 3160.240024] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 3160.575398] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3162.288546] Key type lgssc registered [ 3164.525967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3172.858931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3172.874274] LustreError: 103799:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3173.109479] LustreError: 103815:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3173.124459] LustreError: 103799:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4766 [ 3173.134611] LustreError: 103799:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3173.156759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.165131] LustreError: 103799:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3174.848973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3183.113043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3183.501810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3188.123089] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3193.420795] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing lsmod [ 3196.349176] LustreError: 105020:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3196.357147] LustreError: 105021:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3196.365934] LustreError: 105020:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 3196.373818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3196.374334] LustreError: 105020:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3196.384894] LustreError: 105020:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3196.878478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3202.018859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3202.022494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3202.047671] Lustre: Skipped 1 previous similar message [ 3202.065103] Lustre: Skipped 2 previous similar messages [ 3207.144576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3207.148651] Lustre: Skipped 2 previous similar messages [ 3209.193440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3209.213188] Lustre: Skipped 2 previous similar messages [ 3214.304163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3214.309180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3214.328480] Lustre: Skipped 2 previous similar messages [ 3214.548592] Lustre: server umount lustre-MDT0000 complete [ 3219.071213] LustreError: 103827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596792 with bad export cookie 16664695602721391483 [ 3219.085138] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3219.092420] LustreError: 103827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3219.425844] LustreError: 103847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.436047] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.438617] LustreError: 103847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3219.462161] LustreError: 104283:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000050aba226 x1875479608973056/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3219.465704] Lustre: Skipped 1 previous similar message [ 3219.467211] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3219.721290] Lustre: server umount lustre-MDT0001 complete [ 3226.273708] Lustre: server umount lustre-OST0000 complete [ 3235.621869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.184406] LustreError: 106082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3236.334844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3241.461976] LustreError: 106083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3241.865121] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3246.562231] LustreError: 106082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.649905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3251.685088] LustreError: 106083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.135914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3257.252465] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3265.415948] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3271.742367] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3278.869428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3279.289709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3281.347607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3287.421512] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3295.325442] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3301.859833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3301.860460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3301.868836] LustreError: Skipped 1 previous similar message [ 3301.871571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3301.914904] Lustre: Skipped 1 previous similar message [ 3303.719584] Lustre: server umount lustre-OST0000 complete [ 3308.513591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3308.534521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.884760] Lustre: server umount lustre-MDT0000 complete [ 3316.198971] LustreError: 106082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.224756] LustreError: 106082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3318.602818] LustreError: 106065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596892 with bad export cookie 16664695602721392197 [ 3318.610254] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3318.629462] LustreError: 106065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3319.098884] Lustre: server umount lustre-MDT0001 complete [ 3331.768832] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3335.241468] Key type lgssc unregistered [ 3335.559899] LNet: 109618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3335.579322] LNetError: 109618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3335.608191] LNet: Removed LNI 192.168.204.145@tcp [ 3336.561543] Key type .llcrypt unregistered [ 3336.563826] Key type ._llcrypt unregistered [ 3364.239259] Key type ._llcrypt registered [ 3364.243335] Key type .llcrypt registered [ 3364.382345] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3365.677318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3365.781222] alg: No test for adler32 (adler32-zlib) [ 3367.042143] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 3367.356794] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3369.192201] Key type lgssc registered [ 3370.429786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3380.763463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3380.823983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3382.412548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3387.475710] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3392.994160] LustreError: 111036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.014828] LustreError: 111036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3398.115820] LustreError: 111035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3398.359843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3398.591557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3404.418288] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3414.047563] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3420.674213] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3427.852714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3428.209443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3435.488707] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3445.674786] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3452.793795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3461.467874] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3466.153250] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3473.398506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3473.413403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3473.430366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3473.441739] Lustre: Skipped 1 previous similar message [ 3473.456868] Lustre: Skipped 1 previous similar message [ 3478.498208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3478.503069] Lustre: Skipped 1 previous similar message [ 3483.626885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3483.640887] Lustre: Skipped 1 previous similar message [ 3485.152117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3485.383129] Lustre: server umount lustre-OST0000 complete [ 3490.788158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3490.805334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3490.823049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3493.864871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3496.017315] Lustre: server umount lustre-MDT0000 complete [ 3498.979706] LustreError: 111031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3498.996096] LustreError: 111031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3500.725221] LustreError: 111017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597074 with bad export cookie 12667555386521651186 [ 3500.735455] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3500.744185] LustreError: 111017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3501.387500] Lustre: server umount lustre-MDT0001 complete [ 3515.357977] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3518.621729] Key type lgssc unregistered [ 3519.010413] LNet: 114706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3519.022193] LNetError: 114706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3519.041349] LNet: Removed LNI 192.168.204.145@tcp [ 3519.985817] Key type .llcrypt unregistered [ 3519.988507] Key type ._llcrypt unregistered [ 3539.784061] Key type ._llcrypt registered [ 3539.790115] Key type .llcrypt registered [ 3539.979738] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3541.737814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3541.803603] alg: No test for adler32 (adler32-zlib) [ 3542.990040] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 3543.254313] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3544.954289] Key type lgssc registered [ 3546.653365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3555.190180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3555.215723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.809265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3561.980813] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3567.078934] LustreError: 115924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3567.109456] LustreError: 115924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3571.035348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3571.587248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3576.625517] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3584.390217] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3591.862915] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3600.708447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3601.140671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3602.174074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3608.893337] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3617.800800] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3626.977410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3626.991173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3627.008213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3627.028643] Lustre: Skipped 1 previous similar message [ 3627.049960] Lustre: Skipped 1 previous similar message [ 3628.381633] Lustre: server umount lustre-OST0000 complete [ 3633.121578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3633.136891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3633.149389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3637.220136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3637.223048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3639.518403] Lustre: server umount lustre-MDT0000 complete [ 3642.342808] LustreError: 115924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.367758] LustreError: 115924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3644.054599] LustreError: 117768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597217 with bad export cookie 12997222933603010261 [ 3644.069110] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3644.074771] LustreError: 117768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3644.377950] Lustre: server umount lustre-MDT0001 complete [ 3658.741118] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3661.972362] Key type lgssc unregistered [ 3662.380838] LNet: 119459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3662.388845] LNetError: 119459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3662.403863] LNet: Removed LNI 192.168.204.145@tcp [ 3663.477560] Key type .llcrypt unregistered [ 3663.479564] Key type ._llcrypt unregistered [ 3689.945990] Key type ._llcrypt registered [ 3689.948457] Key type .llcrypt registered [ 3690.070742] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3691.532977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3691.578059] alg: No test for adler32 (adler32-zlib) [ 3692.931899] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 3693.249817] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3695.064299] Key type lgssc registered [ 3696.538830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3704.706748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3704.754910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3706.280427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.032724] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3716.579633] LustreError: 120893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3716.595020] LustreError: 120893:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3720.621308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3721.182290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3726.737365] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3736.230739] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3742.315302] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3749.413229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3749.853679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3751.937823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3757.523791] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3765.771085] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3772.371559] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3775.782890] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3775.787133] Lustre: Skipped 2 previous similar messages [ 3778.662696] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3778.718568] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3780.391219] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3780.401103] Lustre: Skipped 2 previous similar messages [ 3785.766496] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3785.769421] Lustre: Skipped 1 previous similar message [ 3793.219604] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.45@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3793.230403] Lustre: Skipped 2 previous similar messages [ 3795.625188] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.45@tcp 192.168.0.[2,10]@tcp [ 3802.174961] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.45/24@tcp in log lustre-MDT0000 [ 3802.183463] Lustre: Skipped 2 previous similar messages [ 3804.772092] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.45/24@tcp [ 3804.780957] Lustre: Skipped 1 previous similar message [ 3810.808263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3810.809918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.824605] LustreError: Skipped 2 previous similar messages [ 3810.850753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3811.300188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3811.313055] LustreError: Skipped 3 previous similar messages [ 3811.332156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.367290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3811.375846] Lustre: Skipped 1 previous similar message [ 3816.425735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3816.438607] Lustre: Skipped 1 previous similar message [ 3816.616930] Lustre: server umount lustre-OST0000 complete [ 3821.538349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3821.554607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3821.570477] Lustre: Skipped 1 previous similar message [ 3826.658813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3826.668491] Lustre: Skipped 1 previous similar message [ 3835.360186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3835.778864] Lustre: server umount lustre-MDT0000 complete [ 3836.907369] LustreError: 121835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3836.922517] LustreError: 121835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3840.208947] LustreError: 120873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597414 with bad export cookie 14613014039510284707 [ 3840.218393] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3840.233078] LustreError: 120873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3840.700825] Lustre: server umount lustre-MDT0001 complete [ 3853.216781] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3856.749603] Key type lgssc unregistered [ 3857.221775] LNet: 125692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3857.232739] LNetError: 125692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3857.255813] LNet: Removed LNI 192.168.204.145@tcp [ 3858.344930] Key type .llcrypt unregistered [ 3858.352186] Key type ._llcrypt unregistered [ 3893.084645] Key type ._llcrypt registered [ 3893.087688] Key type .llcrypt registered [ 3893.253273] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3894.536054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3894.643222] alg: No test for adler32 (adler32-zlib) [ 3895.911187] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 3896.302642] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3897.960378] Key type lgssc registered [ 3899.711475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3909.301148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3909.318129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.992107] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3911.018618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3911.026857] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.45/24@tcp [ 3916.709739] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3920.366000] LustreError: 127252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3920.406802] LustreError: 127252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3925.477297] LustreError: 127253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3926.965442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3927.307867] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3927.346410] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.45/24@tcp [ 3927.414808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3932.978460] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3941.423576] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3948.236933] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3956.190180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3956.552702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3963.412424] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3971.227357] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3981.192374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3993.578121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3993.596047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3993.617824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3994.305974] Lustre: server umount lustre-OST0000 complete [ 3999.211779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3999.231246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.265331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4001.782084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.783526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4001.828593] Lustre: Skipped 1 previous similar message [ 4006.881447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4006.885366] Lustre: Skipped 1 previous similar message [ 4009.037515] Lustre: server umount lustre-MDT0000 complete [ 4012.028308] LustreError: 127997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4012.058294] LustreError: 127997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4013.593592] LustreError: 127231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597587 with bad export cookie 14004268636642395772 [ 4013.596610] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4013.609928] LustreError: 127231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4014.188371] Lustre: server umount lustre-MDT0001 complete [ 4027.835997] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4031.191804] Key type lgssc unregistered [ 4031.504887] LNet: 130863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4031.516146] LNetError: 130863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4031.533235] LNet: Removed LNI 192.168.204.145@tcp [ 4032.677542] Key type .llcrypt unregistered [ 4032.683177] Key type ._llcrypt unregistered [ 4063.764766] Key type ._llcrypt registered [ 4063.770170] Key type .llcrypt registered [ 4063.923475] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4082.269330] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4083.863029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4083.906036] alg: No test for adler32 (adler32-zlib) [ 4085.167929] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 4085.374874] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4087.048234] Key type lgssc registered [ 4088.604380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4131.244846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4131.284580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4132.646682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4132.707993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4132.849726] Lustre: lustre-MDT0000: new disk, initializing [ 4133.013890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4133.033477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4138.477819] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4151.585709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4151.732990] Lustre: 134263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4151.805886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4151.811453] Lustre: Skipped 1 previous similar message [ 4151.948474] Lustre: lustre-MDT0001: new disk, initializing [ 4152.082286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4152.128474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4152.141589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4157.722525] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4170.152330] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4177.416283] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4185.799459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4186.187524] Lustre: lustre-OST0000: new disk, initializing [ 4186.192144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4186.203684] Lustre: 136236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4186.308899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4192.888062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4192.912085] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4193.143673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4194.504828] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4205.267604] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4210.572974] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4210.806850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4216.409296] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4216.654934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4223.969971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4223.984197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4223.993906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4224.006864] Lustre: Skipped 1 previous similar message [ 4229.096067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4229.102626] Lustre: Skipped 2 previous similar messages [ 4233.184163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4233.402604] Lustre: server umount lustre-OST0000 complete [ 4239.334869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4239.342825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4239.371466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4239.375723] LustreError: Skipped 1 previous similar message [ 4239.402133] Lustre: Skipped 1 previous similar message [ 4244.654288] Lustre: server umount lustre-MDT0000 complete [ 4248.981527] LustreError: 135059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597822 with bad export cookie 323320508536673294 [ 4248.998333] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4249.003578] LustreError: 135059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4249.411292] Lustre: server umount lustre-MDT0001 complete [ 4259.813787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4260.395833] LustreError: 138454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4260.424966] LustreError: 138454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4260.493375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.959789] LustreError: 138455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.123247] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4271.075767] LustreError: 138454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4276.200030] LustreError: 138455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4276.506187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4277.203811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4283.120441] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4291.719128] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4299.028445] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4307.688457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4308.291830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.770508] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4326.056780] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4333.168227] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4333.558740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4339.507835] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4339.956885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4350.916320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4351.180519] Lustre: lustre-OST0001: new disk, initializing [ 4351.186771] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4351.198107] Lustre: 141829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4351.345839] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4356.777395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4356.796107] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4356.953942] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4359.906926] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4370.167726] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4370.352346] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4375.833102] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4376.221458] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4383.106818] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8abec6385800.ost_server_uuid 50 [ 4384.794258] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8abec6385800.ost_server_uuid in IDLE state after 0 sec [ 4398.056101] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.070041] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4398.070798] Lustre: Skipped 1 previous similar message [ 4398.072558] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4398.084718] LustreError: Skipped 1 previous similar message [ 4398.124179] Lustre: Skipped 1 previous similar message [ 4400.400775] Lustre: server umount lustre-OST0001 complete [ 4403.199806] LustreError: 140276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4403.227314] LustreError: 140276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4407.781365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4407.792412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4407.819285] Lustre: Skipped 1 previous similar message [ 4407.827465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4411.517858] Lustre: server umount lustre-OST0000 complete [ 4418.530337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.532905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4418.543233] Lustre: Skipped 2 previous similar messages [ 4418.556275] Lustre: Skipped 3 previous similar messages [ 4430.304117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4430.665116] Lustre: server umount lustre-MDT0000 complete [ 4433.895537] LustreError: 138454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.921896] LustreError: 138454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4435.339741] LustreError: 138436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598009 with bad export cookie 323320508536674176 [ 4435.341866] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4435.363765] LustreError: 138436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4435.963178] Lustre: server umount lustre-MDT0001 complete [ 4458.682151] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4462.017572] Key type lgssc unregistered [ 4462.303588] LNet: 144833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4462.312811] LNetError: 144833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4462.332955] LNet: Removed LNI 192.168.204.145@tcp [ 4463.033392] Key type .llcrypt unregistered [ 4463.043953] Key type ._llcrypt unregistered [ 4507.940607] Key type ._llcrypt registered [ 4507.948040] Key type .llcrypt registered [ 4508.063889] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4523.849448] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4525.042853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4525.189230] alg: No test for adler32 (adler32-zlib) [ 4526.297805] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 4526.632970] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4528.392530] Key type lgssc registered [ 4529.824454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4567.730558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4567.748577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.064431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4569.144284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4569.297405] Lustre: lustre-MDT0000: new disk, initializing [ 4569.410265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4569.434873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4575.176971] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4588.515377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.640733] Lustre: 148987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4588.696501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4588.700647] Lustre: Skipped 1 previous similar message [ 4588.814090] Lustre: lustre-MDT0001: new disk, initializing [ 4588.993913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4589.059436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4589.078868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4594.790520] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4607.523629] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4614.869514] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4623.487904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4623.916254] Lustre: lustre-OST0000: new disk, initializing [ 4623.929912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4623.948981] Lustre: 150959:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4624.089728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4629.598382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4629.607469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4629.802827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4631.612783] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4643.243872] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4648.192736] Lustre: Failing over lustre-OST0000 [ 4649.964381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4649.975841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4650.003825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4650.487695] Lustre: server umount lustre-OST0000 complete [ 4666.858530] Lustre: 147212:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788598224/real 1788598224] req@0000000004175fe8 x1875481041341440/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788598240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4666.887789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4667.938717] Lustre: 147214:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788598224/real 1788598224] req@00000000a0fcdb88 x1875481041341696/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788598240 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4671.398924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4671.991578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4672.019196] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4672.928793] Lustre: 147211:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788598229/real 1788598229] req@00000000d65c44eb x1875481041342592/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788598245 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4677.108745] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4677.586232] Lustre: 147211:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788598234/real 1788598234] req@0000000047f9112c x1875481041343616/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788598250 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4680.798879] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4683.243870] Lustre: 147211:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788598239/real 1788598239] req@0000000035ff9d52 x1875481041344512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788598255 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4686.456814] Lustre: Failing over lustre-MDT0000 [ 4686.503113] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:00 [ 4686.908127] Lustre: server umount lustre-MDT0000 complete [ 4690.271758] LustreError: 148996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4690.295529] LustreError: 148996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4690.317077] Lustre: lustre-OST0000: Recovery over after 0:13, of 3 clients 3 recovered and 0 were evicted. [ 4690.317712] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4690.405281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.313361] LustreError: 150169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4695.339021] LustreError: 150169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4700.435746] LustreError: 151213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4700.466036] LustreError: 151213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4705.557782] LustreError: 148996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4705.604348] LustreError: 148996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4706.593529] Lustre: 147214:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788598264/real 1788598264] req@00000000a8d3c1a8 x1875481041355520/t0(0) o400->MGC192.168.204.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788598280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4706.635244] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4708.948735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4710.682414] LustreError: 151213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4710.704278] LustreError: 151213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4717.044096] LustreError: 147210:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009a28cd52 x1875481041361408/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4717.410844] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4717.526947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4717.603349] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4720.910996] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4722.675479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4722.722225] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4722.817509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4723.816726] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4736.482927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4736.506972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4736.536195] Lustre: Skipped 2 previous similar messages [ 4736.560394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4738.017857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4739.791709] Lustre: server umount lustre-OST0000 complete [ 4748.258458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4748.260470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4748.285991] Lustre: Skipped 1 previous similar message [ 4748.295573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4748.310044] Lustre: Skipped 1 previous similar message [ 4752.208824] Lustre: server umount lustre-MDT0000 complete [ 4753.387331] LustreError: 148996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4753.424493] LustreError: 148996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4757.405120] LustreError: 149793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598331 with bad export cookie 15867440596656813968 [ 4757.424337] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4758.408398] Lustre: server umount lustre-MDT0001 complete [ 4775.781256] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4780.282759] Key type lgssc unregistered [ 4780.758200] LNet: 154620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4780.768534] LNetError: 154620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4780.801873] LNet: Removed LNI 192.168.204.145@tcp [ 4782.208081] Key type .llcrypt unregistered [ 4782.213114] Key type ._llcrypt unregistered [ 4826.102632] Key type ._llcrypt registered [ 4826.107415] Key type .llcrypt registered [ 4826.208948] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4827.422624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4827.477740] alg: No test for adler32 (adler32-zlib) [ 4829.253460] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 4829.603903] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4831.458962] Key type lgssc registered [ 4833.484757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4843.961654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4843.979841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4845.720239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4851.756904] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4855.271277] LustreError: 156138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4855.303956] LustreError: 156138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4860.389595] LustreError: 156139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4863.766169] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4865.507458] LustreError: 156138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4868.606163] LustreError: 156167:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4868.636723] LustreError: 156167:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4868.654081] LustreError: 156167:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 4869.459845] Lustre: server umount lustre-MDT0000 complete [ 4889.876501] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4903.820907] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4947.275183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4947.688371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4947.800518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4948.002812] Lustre: lustre-MDT0000: new disk, initializing [ 4948.340598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4948.379262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4954.381213] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4967.501674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4967.713279] Lustre: 159477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4967.856524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4967.865901] Lustre: Skipped 1 previous similar message [ 4967.946324] Lustre: lustre-MDT0001: new disk, initializing [ 4968.114087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4968.170568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4968.187635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4974.279448] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4986.540597] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4993.682768] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5002.480200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5003.075640] Lustre: lustre-OST0000: new disk, initializing [ 5003.097305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5003.122406] Lustre: 161448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5003.344678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5011.778824] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5013.185437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5013.211058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5013.440510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5024.928380] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5030.968685] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5033.086561] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 04:56:44 (1788598604) [ 5318.625372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5318.632710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5318.642546] Lustre: Skipped 1 previous similar message [ 5318.658757] Lustre: Skipped 2 previous similar messages [ 5323.781551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5323.797302] Lustre: Skipped 2 previous similar messages [ 5324.729648] Lustre: server umount lustre-MDT0000 complete [ 5328.827583] LustreError: 159469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598902 with bad export cookie 18283818694655705566 [ 5328.830114] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5328.846711] LustreError: 159469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5328.870325] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5328.890482] Lustre: Skipped 1 previous similar message [ 5328.892653] LustreError: 161840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.903276] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5333.994359] LustreError: 159484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5333.997170] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5339.105549] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5343.200160] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5343.575751] Lustre: server umount lustre-MDT0001 complete [ 5353.938645] Lustre: server umount lustre-OST0000 complete [ 5364.724210] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 5377.599747] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5420.725834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5421.176400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5421.236708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5421.456061] Lustre: lustre-MDT0000: new disk, initializing [ 5421.652519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5421.679799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5427.754516] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5440.450144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5440.538672] Lustre: 166131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5440.577928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5440.590158] Lustre: Skipped 1 previous similar message [ 5440.655765] Lustre: lustre-MDT0001: new disk, initializing [ 5440.731875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5440.785745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5440.796477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5446.289610] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5459.246421] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5466.925501] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5474.848275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5475.448610] Lustre: lustre-OST0000: new disk, initializing [ 5475.463237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5475.476673] Lustre: 168103:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5475.621966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5482.338937] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5484.632527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5484.642427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5484.811494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5493.598325] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5498.697304] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5499.047425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5503.800326] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5504.049234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5510.628409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5510.645888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5510.651308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5510.668202] Lustre: Skipped 1 previous similar message [ 5510.695182] Lustre: Skipped 1 previous similar message [ 5512.501837] Lustre: server umount lustre-OST0000 complete [ 5517.794663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5517.817522] LustreError: Skipped 1 previous similar message [ 5517.826551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5523.657329] Lustre: server umount lustre-MDT0000 complete [ 5525.988710] LustreError: 168347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5526.019424] LustreError: 168347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5527.961315] LustreError: 166928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599101 with bad export cookie 18283818694655864396 [ 5527.969241] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5527.971168] LustreError: 166928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5528.519270] Lustre: server umount lustre-MDT0001 complete [ 5537.790456] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:05:09 (1788599109) [ 5554.352937] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 5566.718570] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5606.048882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5606.343209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5606.369620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5606.434899] Lustre: lustre-MDT0000: new disk, initializing [ 5606.507422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5606.518924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5610.412864] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5622.279823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5622.511201] Lustre: 172860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5622.920672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5628.722895] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5638.812471] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5645.936224] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5653.548388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5653.849070] Lustre: lustre-OST0000: new disk, initializing [ 5653.853050] Lustre: Skipped 1 previous similar message [ 5653.855932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5653.869790] Lustre: Skipped 2 previous similar messages [ 5653.883939] Lustre: 174838:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5660.996377] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5663.341690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5663.349337] Lustre: Skipped 1 previous similar message [ 5663.356161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5663.478026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5670.709519] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5679.074698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5679.080303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5679.103042] LustreError: Skipped 1 previous similar message [ 5679.107851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5679.107860] Lustre: Skipped 3 previous similar messages [ 5679.170026] Lustre: Skipped 2 previous similar messages [ 5684.564716] Lustre: server umount lustre-OST0000 complete [ 5689.321023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5689.322359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5689.358592] Lustre: Skipped 1 previous similar message [ 5703.136147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5703.388635] Lustre: server umount lustre-MDT0000 complete [ 5704.673300] LustreError: 172871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5704.693548] LustreError: 172871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5707.419882] LustreError: 172852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599281 with bad export cookie 18283818694655865278 [ 5707.438684] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5707.439442] LustreError: 172852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5707.759258] Lustre: server umount lustre-MDT0001 complete [ 5716.498641] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:08:08 (1788599288) [ 5734.477484] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 5745.509109] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5786.592920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5786.888241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5786.920476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5787.006321] Lustre: lustre-MDT0000: new disk, initializing [ 5787.130663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5787.138605] Lustre: Skipped 2 previous similar messages [ 5787.169421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5792.748408] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5804.735328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5804.858030] Lustre: 179165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5805.134098] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5809.948522] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5819.979403] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5826.094647] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5833.089465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5833.439256] Lustre: 181143:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5839.784674] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5839.963723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5840.123971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5849.737340] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5860.834845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5860.866102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5860.872482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5860.879684] Lustre: Skipped 10 previous similar messages [ 5863.994988] Lustre: server umount lustre-OST0000 complete [ 5882.336260] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5882.476730] Lustre: server umount lustre-MDT0000 complete [ 5885.418661] LustreError: 179172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5885.449506] LustreError: 179172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5886.310331] LustreError: 179157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599460 with bad export cookie 18283818694655866363 [ 5886.313360] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5886.322650] LustreError: 179157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5886.717664] Lustre: server umount lustre-MDT0001 complete [ 5900.766848] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5903.840104] Key type lgssc unregistered [ 5904.089611] LNet: 183001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5904.105333] LNetError: 183001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5904.118847] LNet: Removed LNI 192.168.204.145@tcp [ 5904.934158] Key type .llcrypt unregistered [ 5904.936534] Key type ._llcrypt unregistered [ 5930.027204] Key type ._llcrypt registered [ 5930.039303] Key type .llcrypt registered [ 5930.181253] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5931.528918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5931.569171] alg: No test for adler32 (adler32-zlib) [ 5932.700189] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 5933.056769] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5934.888177] Key type lgssc registered [ 5936.609314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5946.665142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5946.698747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5948.338058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5953.472185] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5958.631307] LustreError: 184417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5958.654815] LustreError: 184417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5962.703525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5963.178052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5968.047693] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5975.413635] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5980.675198] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5987.427469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5987.755892] LustreError: 186236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5987.918914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5993.988636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5994.668541] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6003.537671] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6023.651070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6023.658902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6023.688444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6024.677959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6024.686355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6024.689939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.957500] Lustre: server umount lustre-OST0000 complete [ 6034.916637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6034.920841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6034.922090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6034.922096] Lustre: Skipped 1 previous similar message [ 6034.943800] Lustre: Skipped 1 previous similar message [ 6040.033541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6040.040889] Lustre: Skipped 2 previous similar messages [ 6045.155194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6047.200178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6047.456620] Lustre: server umount lustre-MDT0000 complete [ 6049.255172] LustreError: 184416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6049.274639] LustreError: 184416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6051.004632] LustreError: 184396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599624 with bad export cookie 9026634751951919180 [ 6051.009523] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6051.018224] LustreError: 184396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6051.444446] Lustre: server umount lustre-MDT0001 complete [ 6062.946984] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6065.529384] Key type lgssc unregistered [ 6065.790434] LNet: 187971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6065.798124] LNetError: 187971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6065.810219] LNet: Removed LNI 192.168.204.145@tcp [ 6066.784649] Key type .llcrypt unregistered [ 6066.794926] Key type ._llcrypt unregistered [ 6090.453403] Key type ._llcrypt registered [ 6090.455774] Key type .llcrypt registered [ 6090.598827] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6091.708742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6091.750699] alg: No test for adler32 (adler32-zlib) [ 6092.855243] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 6093.215981] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6094.888242] Key type lgssc registered [ 6096.058805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6103.802699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6103.818675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6105.253926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6109.783340] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6115.814871] LustreError: 189382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6115.854580] LustreError: 189382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6118.297048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6118.837422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6123.500657] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6132.365140] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6139.357387] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6147.291490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6147.910710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6151.988189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6155.571769] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6165.345870] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6172.641434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6172.653130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6172.674916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6173.496770] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 6173.672836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6173.678070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6178.638617] Lustre: server umount lustre-OST0000 complete [ 6186.979437] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8abec4873800.ost_server_uuid 50 [ 6189.116818] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6205.924062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6205.939109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6205.968291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6205.978518] Lustre: Skipped 2 previous similar messages [ 6208.485370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6208.495070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6211.062655] Lustre: server umount lustre-MDT0000 complete [ 6213.604174] LustreError: 189382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6213.640954] LustreError: 189382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6214.631291] LustreError: 189383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6214.665368] LustreError: 189383:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6215.655696] LustreError: 189362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599789 with bad export cookie 17278917457291342764 [ 6215.661737] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6215.671307] LustreError: 189362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6216.413557] Lustre: server umount lustre-MDT0001 complete [ 6226.576482] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 05:16:38 (1788599798) [ 6234.432953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.192832] LustreError: 193299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.354655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.737972] LustreError: 193300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6240.876720] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6249.900945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6250.277141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6255.575445] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6264.260525] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6272.399748] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6280.488691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6280.843444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6286.559571] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6294.935344] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6303.241802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6303.553716] Lustre: lustre-OST0001: new disk, initializing [ 6303.563821] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6303.569406] Lustre: 196142:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6303.743995] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6305.811387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6311.092380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6311.105832] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6311.315970] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6311.911741] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6323.321854] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6326.765935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6326.781593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6326.797703] Lustre: Skipped 1 previous similar message [ 6331.362312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6332.681655] Lustre: server umount lustre-OST0000 complete [ 6334.228067] LustreError: 195122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6334.248601] LustreError: 195122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6340.450166] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6340.777420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6346.357708] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6346.775350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6350.317282] LustreError: 195123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6350.342067] LustreError: 195123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6362.597569] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6362.598574] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6362.611671] LustreError: Skipped 1 previous similar message [ 6362.612381] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6362.612385] Lustre: Skipped 3 previous similar messages [ 6362.665471] Lustre: Skipped 2 previous similar messages [ 6367.967079] Lustre: server umount lustre-OST0001 complete [ 6373.358712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6379.199727] Lustre: server umount lustre-MDT0000 complete [ 6383.773677] LustreError: 193279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599957 with bad export cookie 17278917457291343835 [ 6383.778888] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6383.795390] LustreError: 193279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6384.156192] Lustre: server umount lustre-MDT0001 complete [ 6412.130339] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 05:19:43 (1788599983) [ 6418.483174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6418.943163] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6418.974853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6419.118837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6424.149014] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6434.902706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6435.027970] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6435.058683] Lustre: 199583:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6440.834733] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6448.013775] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6453.827857] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6460.002472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6460.172563] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6466.257692] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6467.716214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6473.879676] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6481.169186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6481.243401] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6488.944823] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6497.780603] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6503.393384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6503.407913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6503.422669] Lustre: Skipped 2 previous similar messages [ 6503.430866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6503.433148] Lustre: Skipped 6 previous similar messages [ 6507.384094] Lustre: server umount lustre-OST0000 complete [ 6508.816061] LustreError: 201431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6508.837174] LustreError: 201431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6529.671506] Lustre: server umount lustre-OST0001 complete [ 6536.169204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6536.170570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6536.182576] Lustre: Skipped 4 previous similar messages [ 6536.195031] Lustre: Skipped 6 previous similar messages [ 6540.490409] Lustre: server umount lustre-MDT0000 complete [ 6544.057871] LustreError: 201451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600117 with bad export cookie 17278917457291345242 [ 6544.061765] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6544.073063] LustreError: 201451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6544.440916] Lustre: server umount lustre-MDT0001 complete [ 6569.759344] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 05:22:21 (1788600141) [ 6587.449601] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 6597.500604] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6634.624336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6634.866157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6634.930268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6635.047313] Lustre: lustre-MDT0000: new disk, initializing [ 6635.216860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6635.225393] Lustre: Skipped 3 previous similar messages [ 6635.256133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6639.976223] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6648.990219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6649.115717] Lustre: 207677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6649.197574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6649.201320] Lustre: Skipped 1 previous similar message [ 6649.278314] Lustre: lustre-MDT0001: new disk, initializing [ 6649.388070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6649.396551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6653.644981] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6662.485562] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6668.225581] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6674.538547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6674.868773] Lustre: lustre-OST0000: new disk, initializing [ 6674.871947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6674.881468] Lustre: 209645:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6681.430429] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6682.189617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6682.204890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6682.334787] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6691.201909] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6696.382929] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6696.637405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6701.852086] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6702.259765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6708.196826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6708.199119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6708.205659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6708.205667] Lustre: Skipped 1 previous similar message [ 6708.207991] LustreError: Skipped 3 previous similar messages [ 6708.217151] Lustre: Skipped 1 previous similar message [ 6710.758666] Lustre: server umount lustre-OST0000 complete [ 6729.184189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6729.511393] Lustre: server umount lustre-MDT0000 complete [ 6733.215213] LustreError: 208475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600307 with bad export cookie 17278917457291346803 [ 6733.218735] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6733.221656] LustreError: 208475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6733.630549] Lustre: server umount lustre-MDT0001 complete [ 6741.886372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6742.458317] LustreError: 211864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6742.485621] LustreError: 211864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6747.091602] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6756.216503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6761.519318] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6770.281835] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6776.098376] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6783.269839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6783.655703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6783.660347] Lustre: Skipped 4 previous similar messages [ 6790.043273] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6798.107248] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6802.538198] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6802.759785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6807.260874] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6807.520094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6812.651333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6812.660127] LustreError: Skipped 1 previous similar message [ 6815.577456] Lustre: server umount lustre-OST0000 complete [ 6822.464228] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6827.957549] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6857.005185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6863.600585] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6870.783708] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6875.651648] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6877.949170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6877.975314] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6877.979138] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6877.987293] Lustre: Skipped 1 previous similar message [ 6883.298652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6883.302694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6883.313524] Lustre: Skipped 5 previous similar messages [ 6883.328396] Lustre: Skipped 11 previous similar messages [ 6885.880601] Lustre: server umount lustre-OST0000 complete [ 6896.386483] Lustre: server umount lustre-MDT0000 complete [ 6898.665895] LustreError: 211865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6898.704300] LustreError: 211865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6901.119445] LustreError: 213705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600474 with bad export cookie 17278917457291347685 [ 6901.128141] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6901.139206] LustreError: 213705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6901.605658] Lustre: server umount lustre-MDT0001 complete [ 6911.046919] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 05:28:03 (1788600483) [ 6917.971559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6923.884495] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6934.717177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6940.179466] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6947.049912] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6953.273732] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6961.118680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6968.848644] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6977.300292] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6983.587566] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6983.829059] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6989.327232] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6989.708356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6997.891262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6998.165783] Lustre: lustre-OST0001: new disk, initializing [ 6998.175260] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6998.194331] Lustre: 221199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7004.736883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7004.747845] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7004.901368] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7005.011355] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7015.121244] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7020.986460] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7021.218376] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7026.963360] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7027.227040] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7030.768159] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7030.781158] LustreError: Skipped 3 previous similar messages [ 7035.354125] Lustre: server umount lustre-OST0001 complete [ 7042.883555] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7043.183790] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7048.337113] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7048.556281] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7078.726577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7078.975443] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7078.984513] Lustre: Skipped 5 previous similar messages [ 7084.031772] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7084.054616] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7084.054994] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7085.999431] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7094.040780] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7096.038439] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7104.211550] Lustre: server umount lustre-OST0001 complete [ 7128.431352] Lustre: server umount lustre-OST0000 complete [ 7139.628412] Lustre: server umount lustre-MDT0000 complete [ 7144.131053] LustreError: 218564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600717 with bad export cookie 17278917457291348651 [ 7144.136566] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7144.147445] LustreError: 218564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7150.967934] Lustre: server umount lustre-MDT0001 complete [ 7177.196443] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 05:32:29 (1788600749) [ 7184.154199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7184.498234] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7184.569669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7189.989175] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7200.121090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7200.248487] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7200.296824] Lustre: 226442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7205.333636] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7212.739768] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7218.914690] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7226.488297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7226.689140] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7234.185354] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7242.671838] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7252.519528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7252.605773] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7259.629971] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7267.688781] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7269.860546] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7275.272551] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7275.519940] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7280.850281] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7281.128444] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7287.513576] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8abed23b0000.ost_server_uuid 50 [ 7289.057724] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8abed23b0000.ost_server_uuid in IDLE state after 0 sec [ 7289.973367] Lustre: Permanently deactivating lustre-OST0001 [ 7289.990210] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7293.509536] Lustre: Permanently reactivating lustre-OST0001 [ 7300.079032] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7300.083273] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7300.093262] LustreError: Skipped 5 previous similar messages [ 7300.094606] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7300.094613] Lustre: Skipped 13 previous similar messages [ 7300.137310] Lustre: Skipped 11 previous similar messages [ 7303.722251] Lustre: server umount lustre-OST0001 complete [ 7305.208429] LustreError: 228289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7305.233081] LustreError: 228289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 7314.088782] Lustre: server umount lustre-OST0000 complete [ 7324.657792] Lustre: server umount lustre-MDT0000 complete [ 7328.538331] LustreError: 226436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600902 with bad export cookie 17278917457291349939 [ 7328.539942] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7328.550297] LustreError: 226436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7328.975603] Lustre: server umount lustre-MDT0001 complete [ 7352.677674] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 05:35:24 (1788600924) [ 7360.811208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7361.170671] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7361.225417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7361.268394] Lustre: Skipped 5 previous similar messages [ 7367.670507] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7378.210915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7378.359619] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7378.399039] Lustre: 233138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7383.041592] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7389.803633] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7394.819448] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7401.475975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7401.602983] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7401.619710] Lustre: Permanently deactivating lustre-OST0000 [ 7401.627256] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7408.774834] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7411.253256] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7411.354740] LustreError: 235479:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 7416.424095] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7422.382583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7422.471500] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7428.226836] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7429.510150] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7437.689453] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7439.822654] Lustre: Permanently reactivating lustre-OST0000 [ 7439.841623] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7439.846512] Lustre: Skipped 2 previous similar messages [ 7445.560691] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7445.605598] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7445.615684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7445.631860] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7445.640115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7445.642036] Lustre: Skipped 1 previous similar message [ 7461.326274] Lustre: server umount lustre-OST0001 complete [ 7470.847537] Lustre: server umount lustre-OST0000 complete [ 7481.226403] Lustre: server umount lustre-MDT0000 complete [ 7484.867859] LustreError: 233132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601058 with bad export cookie 17278917457291351612 [ 7484.869661] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7484.881794] LustreError: 233132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7485.363972] Lustre: server umount lustre-MDT0001 complete [ 7496.174429] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7499.308928] Key type lgssc unregistered [ 7499.588748] LNet: 238016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7499.598873] LNetError: 238016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7499.616085] LNet: Removed LNI 192.168.204.145@tcp [ 7500.364628] Key type .llcrypt unregistered [ 7500.367432] Key type ._llcrypt unregistered [ 7522.575461] Key type ._llcrypt registered [ 7522.579218] Key type .llcrypt registered [ 7522.669556] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7523.370398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7523.441803] alg: No test for adler32 (adler32-zlib) [ 7524.512466] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 7524.824525] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7526.512346] Key type lgssc registered [ 7527.706720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7536.603760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7536.648174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7538.109025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7542.210449] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7547.368340] LustreError: 239479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.384959] LustreError: 239479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7550.243713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7550.345941] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7550.370779] Lustre: 239466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7550.384682] Lustre: Permanently deactivating lustre-MDT0001 [ 7550.399917] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7550.690588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7554.943786] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7556.667751] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7562.972939] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7569.531000] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7576.296660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7576.550268] LustreError: 241318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7576.669619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7581.676120] LustreError: 241319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7581.698678] LustreError: 241319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7581.722044] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7582.712418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7582.941691] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7587.817090] LustreError: 241318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7587.842828] LustreError: 241318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7591.173113] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7592.944284] LustreError: 241319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7592.967932] LustreError: 241319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7598.827485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7599.051559] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7604.214044] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7604.224280] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7605.862920] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7614.361554] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7618.550625] Lustre: Permanently reactivating lustre-MDT0001 [ 7618.561930] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7618.567610] Lustre: Skipped 2 previous similar messages [ 7625.012367] Lustre: lustre-MDT0001: Client 6e3ea558-bf67-4298-be68-bdb640405b0f (at 192.168.204.45@tcp) reconnecting [ 7625.349124] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7625.361357] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7625.371195] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7625.385444] LustreError: 243047:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7625.388534] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7625.404222] LustreError: 243047:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7625.424204] LustreError: 243047:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.145@tcp: cfg command failed: rc = -108 [ 7625.434315] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7625.434315] [ 7625.448439] LustreError: 239471:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7631.130154] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7632.735192] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7638.852394] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7639.087654] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7644.388532] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7644.709906] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7647.757895] Lustre: Permanently deactivating lustre-MDT0001 [ 7647.781922] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7647.785685] Lustre: Skipped 2 previous similar messages [ 7655.580663] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7663.588718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7663.598255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7663.623004] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7664.454469] Lustre: server umount lustre-MDT0000 complete [ 7665.636208] LustreError: 239478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7665.641268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7665.672453] LustreError: 239478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7665.688771] Lustre: Skipped 1 previous similar message [ 7668.550807] LustreError: 239458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601242 with bad export cookie 15123249583281386085 [ 7668.559388] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7668.568335] LustreError: 239458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7668.924444] Lustre: server umount lustre-MDT0001 complete [ 7675.053609] Lustre: server umount lustre-OST0000 complete [ 7687.178801] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:40:59 (1788601259) [ 7698.267328] Lustre: server umount lustre-OST0001 complete [ 7705.388704] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 7713.562730] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7753.921562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7754.070252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7754.079480] Lustre: Skipped 2 previous similar messages [ 7754.099393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7754.196384] Lustre: lustre-MDT0000: new disk, initializing [ 7754.284274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7754.302581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7757.958695] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7768.306022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7768.441258] Lustre: 247737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7768.515806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7768.523197] Lustre: Skipped 1 previous similar message [ 7768.688931] Lustre: lustre-MDT0001: new disk, initializing [ 7768.818504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7768.883933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7768.914468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7774.326170] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7785.994986] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7791.452825] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7798.148692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7798.485396] Lustre: lustre-OST0000: new disk, initializing [ 7798.491646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7798.500331] Lustre: 249708:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7798.645541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7803.968338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7803.975535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7804.141524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7805.208160] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7815.005570] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7817.219493] LustreError: 249310:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7826.443598] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7826.618988] Lustre: lustre-OST0001: new disk, initializing [ 7826.622926] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7826.631688] Lustre: 250820:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7827.240125] LustreError: 249310:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7827.273976] LustreError: 247743:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7833.491612] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7836.264323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7836.271706] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7836.426205] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7837.368318] LustreError: 247743:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7843.582171] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7845.568314] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7852.007596] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7852.008468] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7852.017975] LustreError: Skipped 1 previous similar message [ 7852.024148] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7852.024893] Lustre: Skipped 2 previous similar messages [ 7852.043897] Lustre: Skipped 1 previous similar message [ 7853.625159] Lustre: server umount lustre-OST0001 complete [ 7857.123656] LustreError: 249690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7857.157101] LustreError: 249690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7870.945598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7870.946533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7870.951882] LustreError: Skipped 1 previous similar message [ 7870.960232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7870.966814] Lustre: Skipped 1 previous similar message [ 7875.703210] Lustre: server umount lustre-OST0000 complete [ 7881.702468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7881.721347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7881.746937] Lustre: Skipped 1 previous similar message [ 7881.752937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7881.758638] Lustre: Skipped 2 previous similar messages [ 7886.745697] Lustre: server umount lustre-MDT0000 complete [ 7891.282972] LustreError: 247729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601465 with bad export cookie 15123249583281388892 [ 7891.293577] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7891.298979] LustreError: 247729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7897.893476] Lustre: server umount lustre-MDT0001 complete [ 7908.615481] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7911.261794] Key type lgssc unregistered [ 7911.620196] LNet: 252853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7911.628852] LNetError: 252853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7911.649556] LNet: Removed LNI 192.168.204.145@tcp [ 7912.506116] Key type .llcrypt unregistered [ 7912.509354] Key type ._llcrypt unregistered [ 7951.873817] Key type ._llcrypt registered [ 7951.882306] Key type .llcrypt registered [ 7952.022606] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7953.468984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7953.534498] alg: No test for adler32 (adler32-zlib) [ 7954.719357] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 7955.044817] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7956.752208] Key type lgssc registered [ 7957.810806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7965.654971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7965.671825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7966.941599] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7966.959533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7967.121182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7972.424343] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7981.605158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7981.769792] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7981.825968] Lustre: 255060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7982.075046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7987.242453] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7993.903829] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7999.137260] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8006.545288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8006.706097] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8006.879568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8012.977788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8013.937400] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8023.051340] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8074.215989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8074.223733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8074.246509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8074.726914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8074.728952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8079.846117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8079.851956] Lustre: Skipped 2 previous similar messages [ 8080.430508] Lustre: server umount lustre-OST0000 complete [ 8085.201534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8099.783232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8100.340400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8104.435234] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8104.443949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8104.450777] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8108.012588] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8117.625474] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8160.742297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8160.745730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8160.759823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8160.798973] Lustre: Skipped 1 previous similar message [ 8164.322595] Lustre: server umount lustre-OST0000 complete [ 8170.979167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8170.997918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8171.010954] Lustre: Skipped 2 previous similar messages [ 8174.750789] Lustre: server umount lustre-MDT0000 complete [ 8178.685819] LustreError: 255054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601752 with bad export cookie 14902056339686474172 [ 8178.695793] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8178.697110] LustreError: 255054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8179.234918] Lustre: server umount lustre-MDT0001 complete [ 8192.162294] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8194.564375] Key type lgssc unregistered [ 8194.914963] LNet: 260491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8194.925162] LNetError: 260491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8194.951939] LNet: Removed LNI 192.168.204.145@tcp [ 8196.384960] Key type .llcrypt unregistered [ 8196.386553] Key type ._llcrypt unregistered [ 8221.971753] Key type ._llcrypt registered [ 8221.976501] Key type .llcrypt registered [ 8222.073243] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8223.379740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8223.408812] alg: No test for adler32 (adler32-zlib) [ 8224.490379] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 8224.720608] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8226.448125] Key type lgssc registered [ 8227.924270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8237.160429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8237.198095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8238.882579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8243.209813] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8248.295387] LustreError: 261906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8248.318690] LustreError: 261906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8252.102411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8252.608556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8258.167176] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8265.900930] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8272.161671] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8280.121351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8280.552476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8284.686927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8288.959536] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8298.034802] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8325.601661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8325.611778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8325.633354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8328.681446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8328.689049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8328.690364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8330.141562] Lustre: server umount lustre-OST0000 complete [ 8338.915550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8338.921332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8338.935842] Lustre: Skipped 1 previous similar message [ 8338.967750] Lustre: Skipped 2 previous similar messages [ 8344.033553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8344.046033] Lustre: Skipped 1 previous similar message [ 8349.152140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8349.154401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8349.184764] Lustre: Skipped 1 previous similar message [ 8349.403931] Lustre: server umount lustre-MDT0000 complete [ 8353.410379] LustreError: 265260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601927 with bad export cookie 962652355996290803 [ 8353.420123] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8353.753852] Lustre: server umount lustre-MDT0001 complete [ 8366.329623] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8369.604228] Key type lgssc unregistered [ 8369.951648] LNet: 266215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8369.991783] LNetError: 266215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8371.047120] LNet: Removed LNI 192.168.204.145@tcp [ 8371.942685] Key type .llcrypt unregistered [ 8371.945075] Key type ._llcrypt unregistered [ 8391.625728] Key type ._llcrypt registered [ 8391.633119] Key type .llcrypt registered [ 8391.762760] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8392.921249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8393.067117] alg: No test for adler32 (adler32-zlib) [ 8394.110902] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 8394.441490] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8396.129064] Key type lgssc registered [ 8397.547432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8405.991577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8406.020704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8407.659921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8413.160763] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8417.762297] LustreError: 267433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8417.781672] LustreError: 267433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8422.348701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8422.698097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8427.456827] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8434.787356] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8440.738323] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8447.271444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8447.463725] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 8447.552230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8453.156303] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8453.620625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8462.595884] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8478.177246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8478.185703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8478.206971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8479.218655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8479.242224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8479.243410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8479.263800] Lustre: Skipped 1 previous similar message [ 8481.691427] Lustre: server umount lustre-OST0000 complete [ 8489.446116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8489.446282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8489.456174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8489.462797] Lustre: Skipped 1 previous similar message [ 8492.337649] Lustre: server umount lustre-MDT0000 complete [ 8494.563894] LustreError: 268162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8494.596196] LustreError: 268162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8496.132197] LustreError: 267414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602069 with bad export cookie 701658831894351537 [ 8496.153570] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8496.158366] LustreError: 267414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8496.517178] Lustre: server umount lustre-MDT0001 complete [ 8507.824667] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8511.080934] Key type lgssc unregistered [ 8511.425131] LNet: 271108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8511.439441] LNetError: 271108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8511.461536] LNet: Removed LNI 192.168.204.145@tcp [ 8512.461541] Key type .llcrypt unregistered [ 8512.463454] Key type ._llcrypt unregistered [ 8534.873156] Key type ._llcrypt registered [ 8534.875603] Key type .llcrypt registered [ 8534.983748] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8536.555280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8536.617100] alg: No test for adler32 (adler32-zlib) [ 8537.745846] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 8538.043233] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8539.768224] Key type lgssc registered [ 8541.082391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8548.982932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8549.007324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8550.452320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8554.980682] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8560.620562] LustreError: 272520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8560.638670] LustreError: 272520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8563.485055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8563.791390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8568.113632] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8575.461415] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8581.208462] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8588.089834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8588.198754] LustreError: 274342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8588.511046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8594.638492] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8595.967685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8602.946256] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8626.661226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8626.664160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8626.665131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8626.719717] Lustre: Skipped 1 previous similar message [ 8631.777207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8631.789799] Lustre: Skipped 2 previous similar messages [ 8632.556257] Lustre: server umount lustre-OST0000 complete [ 8636.897405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8636.917804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8636.929369] Lustre: Skipped 1 previous similar message [ 8642.019372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8642.025834] Lustre: Skipped 1 previous similar message [ 8642.687467] Lustre: server umount lustre-MDT0000 complete [ 8646.634518] LustreError: 272501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602220 with bad export cookie 1062106911300461470 [ 8646.640171] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8646.646263] LustreError: 272501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8647.010810] Lustre: server umount lustre-MDT0001 complete [ 8659.992636] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8663.351657] Key type lgssc unregistered [ 8663.739422] LNet: 276828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8663.747765] LNetError: 276828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8663.761304] LNet: Removed LNI 192.168.204.145@tcp [ 8664.708564] Key type .llcrypt unregistered [ 8664.713827] Key type ._llcrypt unregistered [ 8682.843432] Key type ._llcrypt registered [ 8682.845274] Key type .llcrypt registered [ 8682.929354] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8684.123543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8684.349748] alg: No test for adler32 (adler32-zlib) [ 8685.523717] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 8685.855328] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8687.592166] Key type lgssc registered [ 8688.933917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8697.342865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8697.369861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8699.100737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8704.703423] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8708.580051] LustreError: 278059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8708.598221] LustreError: 278059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8713.697759] LustreError: 278061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.878665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8715.259167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8720.798409] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8729.417239] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8735.550849] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8743.493537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8743.975127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8749.054499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8751.472512] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8759.145460] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8774.625624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8774.638934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8774.671477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8775.652519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8775.656362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8775.669345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8778.530548] Lustre: server umount lustre-OST0000 complete [ 8785.899232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8785.922704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8785.925031] Lustre: Skipped 1 previous similar message [ 8785.945440] Lustre: Skipped 2 previous similar messages [ 8789.500816] Lustre: server umount lustre-MDT0000 complete [ 8791.012489] LustreError: 278046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8791.032740] LustreError: 278046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8793.575205] LustreError: 278026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602367 with bad export cookie 4085018150756608555 [ 8793.587387] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8793.591945] LustreError: 278026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8794.123157] Lustre: server umount lustre-MDT0001 complete [ 8807.267625] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8810.652553] Key type lgssc unregistered [ 8811.069608] LNet: 281761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8811.076279] LNetError: 281761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8811.096217] LNet: Removed LNI 192.168.204.145@tcp [ 8812.206338] Key type .llcrypt unregistered [ 8812.211514] Key type ._llcrypt unregistered [ 8825.132485] Key type ._llcrypt registered [ 8825.141202] Key type .llcrypt registered [ 8825.277294] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8844.546073] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 8861.864248] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8862.994433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8863.122840] alg: No test for adler32 (adler32-zlib) [ 8864.618375] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 8865.106416] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8866.881853] Key type lgssc registered [ 8868.217657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8907.921756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8907.967872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8909.286181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8909.322466] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8909.420213] Lustre: lustre-MDT0000: new disk, initializing [ 8909.496647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8909.519525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8914.957791] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8925.852200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8926.049062] Lustre: 285321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8926.149592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8926.163283] Lustre: Skipped 1 previous similar message [ 8926.304731] Lustre: lustre-MDT0001: new disk, initializing [ 8926.425875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8926.473614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8926.492363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8931.955787] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8941.746874] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8948.336223] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8956.643293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8956.967227] Lustre: lustre-OST0000: new disk, initializing [ 8956.973540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8956.979886] Lustre: 287294:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8957.062749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8962.873673] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8965.187274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8965.193315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8965.290942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8971.943776] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8977.661485] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8978.067955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8984.422510] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8984.909818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8991.207362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8991.217303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8991.218459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8991.247030] Lustre: Skipped 1 previous similar message [ 8993.768066] Lustre: server umount lustre-OST0000 complete [ 8998.384539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8998.390694] LustreError: Skipped 1 previous similar message [ 8998.396956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8998.407328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8998.414489] Lustre: Skipped 1 previous similar message [ 9001.443287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9001.445055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9001.463174] Lustre: Skipped 1 previous similar message [ 9004.367740] Lustre: server umount lustre-MDT0000 complete [ 9007.279789] LustreError: 287292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602581 with bad export cookie 18237875062095126917 [ 9007.284972] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9007.296200] LustreError: 287292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9007.798092] Lustre: server umount lustre-MDT0001 complete [ 9016.821261] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:03:08 (1788602588) [ 9024.043891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9024.513289] LustreError: 289708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9024.544765] LustreError: 289708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9024.619239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9029.538530] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9029.602661] LustreError: 289709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9034.732064] LustreError: 289708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9038.648545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9039.075362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9043.912446] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9052.155830] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9057.789505] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9063.786857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9064.115590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9070.408763] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9078.118023] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9094.128897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9094.146523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9094.196383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9094.208052] Lustre: Skipped 1 previous similar message [ 9099.068206] Lustre: server umount lustre-OST0000 complete [ 9104.354253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9104.378710] Lustre: Skipped 1 previous similar message [ 9104.381562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9109.785383] Lustre: server umount lustre-MDT0000 complete [ 9113.524514] LustreError: 289690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602687 with bad export cookie 18237875062095127792 [ 9113.531442] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9113.555408] LustreError: 289690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9114.172300] Lustre: server umount lustre-MDT0001 complete [ 9126.540716] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 9129.079913] Key type lgssc unregistered [ 9129.313650] LNet: 293263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9129.334555] LNetError: 293263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9129.357390] LNet: Removed LNI 192.168.204.145@tcp [ 9130.375354] Key type .llcrypt unregistered [ 9130.377667] Key type ._llcrypt unregistered [ 9159.328581] Key type ._llcrypt registered [ 9159.333759] Key type .llcrypt registered [ 9159.432424] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 9175.102729] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9176.196051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9176.261160] alg: No test for adler32 (adler32-zlib) [ 9177.402929] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 9177.658658] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 9179.330450] Key type lgssc registered [ 9180.407979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9227.442129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9227.459797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9228.754489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9228.793307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9228.918909] Lustre: lustre-MDT0000: new disk, initializing [ 9229.102820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9229.121349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9233.484572] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9244.420589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9244.537850] Lustre: 297085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9244.595817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9244.600980] Lustre: Skipped 1 previous similar message [ 9244.659322] Lustre: lustre-MDT0001: new disk, initializing [ 9244.747282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9244.830047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9244.848774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9249.944882] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9260.880992] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9267.344285] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9275.517428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9275.973941] Lustre: lustre-OST2710: new disk, initializing [ 9275.979892] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9275.991020] Lustre: 299056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9276.110959] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9281.646064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9281.658631] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9281.837996] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9284.300338] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9295.057898] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9302.803622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9303.054559] Lustre: lustre-OST03e8: new disk, initializing [ 9303.059098] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9303.065993] Lustre: 300124:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9303.194524] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9310.036921] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9311.846904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9311.866991] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9311.985346] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9317.143650] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9324.741443] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9325.039423] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9329.844350] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9330.082487] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9334.807837] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9335.031279] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9339.947125] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9340.174345] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9347.041326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9347.046757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9347.062349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9347.293950] Lustre: server umount lustre-MDT0000 complete [ 9348.072205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9348.083277] Lustre: Skipped 1 previous similar message [ 9350.884151] LustreError: 297079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602924 with bad export cookie 8582837327603900766 [ 9350.886200] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9350.898892] LustreError: 297079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9351.250795] Lustre: server umount lustre-MDT0001 complete [ 9364.614339] Lustre: server umount lustre-OST2710 complete [ 9379.054948] Lustre: server umount lustre-OST03e8 complete [ 9398.773158] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 9412.646618] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9451.129643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9451.374296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9451.397470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9451.503792] Lustre: lustre-MDT0000: new disk, initializing [ 9451.638062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9451.660543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9456.392667] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9466.046691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9466.139267] Lustre: 305376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9466.189724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9466.194682] Lustre: Skipped 1 previous similar message [ 9466.257596] Lustre: lustre-MDT0001: new disk, initializing [ 9466.336781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9466.370379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9466.378582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9470.345660] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9478.878274] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9484.152534] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9489.374602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9489.728985] Lustre: lustre-OST0000: new disk, initializing [ 9489.734569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9489.742508] Lustre: 307342:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9489.879100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9496.926917] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9498.179407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9498.194860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9498.282988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9505.808684] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9511.060849] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9511.291091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9516.737306] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9517.040841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9519.076144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9519.085367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9519.089561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9519.098752] Lustre: Skipped 2 previous similar messages [ 9519.109758] Lustre: Skipped 1 previous similar message [ 9524.220351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9529.318357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9529.327269] Lustre: Skipped 2 previous similar messages [ 9533.408440] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9533.594659] Lustre: server umount lustre-OST0000 complete [ 9538.026471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9538.037504] LustreError: Skipped 1 previous similar message [ 9538.042758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9538.057618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9543.741174] Lustre: server umount lustre-MDT0000 complete [ 9544.675661] LustreError: 305387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9544.702996] LustreError: 305387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9547.126953] LustreError: 305367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603120 with bad export cookie 8582837327603902446 [ 9547.127728] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9547.140651] LustreError: 305367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9547.479858] Lustre: server umount lustre-MDT0001 complete [ 9555.887872] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 06:12:07 (1788603127) [ 9557.777553] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9559.593335] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 06:12:11 (1788603131) [ 9563.126596] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9587.306754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9587.595665] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9587.638932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9587.853368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9593.314076] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9601.383324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9601.547439] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9601.608470] Lustre: 311203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9606.926635] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9626.937485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9627.191669] Lustre: Denying initial registration attempt from nid 192.168.204.145@tcp, specified as failover [ 9627.199123] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9627.204478] LustreError: 312655:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9627.208236] LustreError: 312655:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9627.215849] LustreError: 312655:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9627.342440] Lustre: server umount lustre-OST0000 complete [ 9627.346311] LustreError: 312655:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9635.813338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9635.821769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9635.825932] Lustre: Skipped 2 previous similar messages [ 9635.844230] Lustre: Skipped 3 previous similar messages [ 9648.098219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9648.221915] Lustre: server umount lustre-MDT0000 complete [ 9651.173817] LustreError: 311212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9651.193414] LustreError: 311212:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9651.501800] LustreError: 311194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603225 with bad export cookie 8582837327603903328 [ 9651.505037] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9651.515701] LustreError: 311194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9651.802620] Lustre: server umount lustre-MDT0001 complete [ 9661.620582] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 9664.424018] Key type lgssc unregistered [ 9664.714369] LNet: 313775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9664.723600] LNetError: 313775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9664.737273] LNet: Removed LNI 192.168.204.145@tcp [ 9665.484194] Key type .llcrypt unregistered [ 9665.486285] Key type ._llcrypt unregistered [ 9675.793380] Key type ._llcrypt registered [ 9675.798510] Key type .llcrypt registered [ 9675.878895] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9676.641839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9676.873964] alg: No test for adler32 (adler32-zlib) [ 9677.924879] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 9678.210403] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 9679.928286] Key type lgssc registered [ 9680.944264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9726.984298] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9735.236935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9735.250626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9736.515488] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9736.532544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9736.652364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9741.256293] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9749.779357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9749.981306] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9750.044412] Lustre: 316534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9750.402824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9755.504602] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9764.018121] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9769.634430] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9789.549981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9789.787304] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9789.977742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9796.306432] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9804.298181] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9808.356099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9808.361773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9808.386595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9808.888666] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9808.900317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9808.927423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9812.925354] Lustre: server umount lustre-OST0000 complete [ 9817.057784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9817.072071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9817.088808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9819.111717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9819.115123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9822.833931] Lustre: server umount lustre-MDT0000 complete [ 9827.068054] LustreError: 316528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603400 with bad export cookie 1723485899011234802 [ 9827.071039] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9827.085479] LustreError: 316528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9827.520786] Lustre: server umount lustre-MDT0001 complete [ 9839.503462] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 9842.758348] Key type lgssc unregistered [ 9843.089926] LNet: 320109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9843.097548] LNetError: 320109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9844.135120] LNet: Removed LNI 192.168.204.145@tcp [ 9845.198119] Key type .llcrypt unregistered [ 9845.201494] Key type ._llcrypt unregistered [ 9871.910247] Key type ._llcrypt registered [ 9871.913037] Key type .llcrypt registered [ 9872.082537] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9873.035284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9873.066703] alg: No test for adler32 (adler32-zlib) [ 9874.257104] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 9874.596677] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 9876.288445] Key type lgssc registered [ 9877.388763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9884.856335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9884.869564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9886.397223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9891.152711] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9896.930752] LustreError: 321523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.940844] LustreError: 321523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9899.530479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9899.871240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9904.947852] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9912.496171] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9917.751472] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9926.323131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9926.667582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9933.388911] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9941.082864] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9955.300757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9955.302331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9955.317648] Lustre: Skipped 2 previous similar messages [ 9955.339157] Lustre: Skipped 2 previous similar messages [ 9958.182935] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 9960.426523] LustreError: 322252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9960.446699] LustreError: 322252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9960.509855] Lustre: server umount lustre-MDT0000 complete [ 9963.272029] LustreError: 322958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9964.216325] LustreError: 321504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603538 with bad export cookie 5128728094950879125 [ 9964.224476] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9964.228874] LustreError: 321504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9965.539989] LustreError: 324016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9965.542785] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9965.566550] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9968.419227] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [ 9969.569787] LustreError: 321519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9969.596965] LustreError: 321519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9970.677272] Lustre: server umount lustre-MDT0001 complete [ 9977.844696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9985.577425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9990.348823] LustreError: 323357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9990.406306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9990.454268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9994.951955] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9999.611580] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10004.678161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10004.963855] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10004.979363] Lustre: Skipped 2 previous similar messages [10005.105883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10010.615423] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10010.768583] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10019.523664] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10021.458250] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10026.873161] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10028.304193] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10035.691206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10035.696535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10035.707251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10039.609541] Lustre: server umount lustre-OST0000 complete [10045.923578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10045.936331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10045.948958] Lustre: Skipped 1 previous similar message [10049.713272] Lustre: server umount lustre-MDT0000 complete [10051.562509] LustreError: 325144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10051.593108] LustreError: 325144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10053.391616] LustreError: 324330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603627 with bad export cookie 5128728094950886825 [10053.398538] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10053.408313] LustreError: 324330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10053.725691] Lustre: server umount lustre-MDT0001 complete [10064.428792] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [10067.763930] Key type lgssc unregistered [10068.048349] LNet: 327812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10068.071711] LNetError: 327812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10068.104276] LNet: Removed LNI 192.168.204.145@tcp [10068.942528] Key type .llcrypt unregistered [10068.950810] Key type ._llcrypt unregistered [10091.869246] Key type ._llcrypt registered [10091.875194] Key type .llcrypt registered [10092.024242] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [10092.957511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10092.980532] alg: No test for adler32 (adler32-zlib) [10094.082371] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [10094.327519] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [10096.032183] Key type lgssc registered [10097.005176] Lustre: Echo OBD driver; http://www.lustre.org/ [10104.856619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10104.873540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10106.373881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10110.530662] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10116.580184] LustreError: 329227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10116.607655] LustreError: 329227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10118.925437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10119.350194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10124.129871] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10131.847863] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10138.284929] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10142.180500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10142.217737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10142.225277] Lustre: Skipped 1 previous similar message [10142.249848] Lustre: Skipped 1 previous similar message [10144.298514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10144.307862] Lustre: Skipped 1 previous similar message [10147.979198] Lustre: server umount lustre-MDT0000 complete [10149.349312] LustreError: 329226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10149.364841] LustreError: 329226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10151.169602] LustreError: 329207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603725 with bad export cookie 13079481440619139487 [10151.183667] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10151.184807] LustreError: 329207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10151.494705] Lustre: server umount lustre-MDT0001 complete [10166.599509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10166.984671] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10167.047562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10167.267058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10171.886639] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10179.932800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10180.067410] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10180.118609] Lustre: 331815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10180.398938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10184.966458] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10192.423729] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10198.172649] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10207.541956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10207.768943] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10207.783068] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10207.803531] LustreError: 333697:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10207.818080] LustreError: 333697:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10207.832502] LustreError: 333697:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10207.950660] Lustre: server umount lustre-OST0000 complete [10207.953532] LustreError: 333697:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10218.480416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10218.645610] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10218.916440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10224.423455] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10226.799330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10231.998450] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10239.835637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10240.002811] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10240.090428] Lustre: lustre-OST0001: new disk, initializing [10240.095608] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10240.118396] Lustre: 335240:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10240.282833] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10246.653966] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10249.348254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10249.358174] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10249.474549] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10254.268551] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10259.941750] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10259.960530] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10259.973212] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10263.750365] Lustre: server umount lustre-OST0001 complete [10265.072549] LustreError: 334166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10268.135950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10268.145581] LustreError: Skipped 1 previous similar message [10268.154617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10268.178978] Lustre: Skipped 1 previous similar message [10268.185416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10268.205515] Lustre: Skipped 1 previous similar message [10270.185800] LustreError: 336107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10270.187787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10270.203694] LustreError: 336107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10273.953726] Lustre: server umount lustre-OST0000 complete [10277.857960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10277.871787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10277.895034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10277.900257] Lustre: Skipped 2 previous similar messages [10283.469202] Lustre: server umount lustre-MDT0000 complete [10285.538858] LustreError: 331823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10285.551222] LustreError: 331823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10287.032818] LustreError: 334182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603860 with bad export cookie 13079481440619140040 [10287.036684] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10287.051622] LustreError: 334182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10287.485722] Lustre: server umount lustre-MDT0001 complete [10299.851892] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [10302.145676] Key type lgssc unregistered [10302.441265] LNet: 337194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10302.454210] LNetError: 337194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10303.537047] LNet: Removed LNI 192.168.204.145@tcp [10304.386428] Key type .llcrypt unregistered [10304.392985] Key type ._llcrypt unregistered [10375.285374] Key type ._llcrypt registered [10375.288339] Key type .llcrypt registered [10375.382664] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [10390.157337] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [10391.469510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10391.528443] alg: No test for adler32 (adler32-zlib) [10392.575627] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [10392.810270] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [10394.528176] Key type lgssc registered [10395.271701] Lustre: Echo OBD driver; http://www.lustre.org/ [10429.292543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10429.316700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10430.567845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10430.602414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10430.684396] Lustre: lustre-MDT0000: new disk, initializing [10430.797091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10430.823969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10436.076091] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10447.539434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10447.712388] Lustre: 342126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10447.813777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10447.822114] Lustre: Skipped 1 previous similar message [10447.945868] Lustre: lustre-MDT0001: new disk, initializing [10448.082378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10448.151812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10448.165084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10453.279557] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10462.871918] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10468.530518] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10475.072486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10475.282738] Lustre: lustre-OST0000: new disk, initializing [10475.285675] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10475.289272] Lustre: 344099:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10475.361481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10480.849888] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10482.256894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10482.266894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10482.400674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10489.703944] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10494.609066] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10494.870032] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10499.712798] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10499.922358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10503.149238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10503.156503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10503.157396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10503.157499] LustreError: Skipped 1 previous similar message [10503.208669] Lustre: Skipped 1 previous similar message [10508.145328] Lustre: server umount lustre-OST0000 complete [10513.378613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10513.391695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10513.405241] Lustre: Skipped 1 previous similar message [10513.413578] Lustre: Skipped 1 previous similar message [10517.854190] Lustre: server umount lustre-MDT0000 complete [10521.710626] LustreError: 342120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604095 with bad export cookie 15123550214491526036 [10521.718510] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10521.732501] LustreError: 342120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10522.046925] Lustre: server umount lustre-MDT0001 complete [10531.588666] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 06:28:23 (1788604103) [10539.702425] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 06:28:31 (1788604111) [10548.237846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10548.867682] LustreError: 346852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10548.894093] LustreError: 346852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10548.958727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10554.024523] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10554.341712] LustreError: 346853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10558.437027] LustreError: 346852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10562.287274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10562.575325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10566.156779] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10572.388491] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10578.208325] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10585.694693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10586.037709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10593.290042] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10601.525257] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10609.310765] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10611.165347] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10612.556226] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10614.123904] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10616.002543] Lustre: Failing over lustre-MDT0000 [10616.258571] Lustre: server umount lustre-MDT0000 complete [10618.692744] LustreError: 346849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10618.849936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10618.862027] Lustre: Skipped 2 previous similar messages [10618.877781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10623.751536] LustreError: 346849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10623.778581] LustreError: 346849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10633.997106] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10634.019739] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10634.209170] Lustre: 340350:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788604192/real 1788604192] req@00000000c70ac03c x1875487192322944/t0(0) o400->MGC192.168.204.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788604208 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10634.228935] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10634.855388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10644.463550] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd1e1a4cac58cdb20 [10644.505660] Lustre: MGC192.168.204.145@tcp: Connection restored to 0@lo (at 0@lo) [10644.776180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10644.836777] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10645.261578] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10649.448502] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10650.110545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10650.155996] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10650.230430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10658.539557] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10659.986083] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10663.753381] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10665.738148] Lustre: Failing over lustre-MDT0000 [10665.779267] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [10665.793322] Lustre: Skipped 1 previous similar message [10666.134866] Lustre: server umount lustre-MDT0000 complete [10670.569153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10670.572326] LustreError: 348588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10670.581093] Lustre: Skipped 2 previous similar messages [10670.583176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10670.599757] LustreError: 348588:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [10685.090482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10685.199742] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10685.451340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10685.511425] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10686.221546] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10690.087532] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10690.563315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10690.575723] Lustre: Skipped 2 previous similar messages [10690.601444] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10690.654328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10699.520616] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10700.902139] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10711.016293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10711.017290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10711.028122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10711.034734] Lustre: Skipped 1 previous similar message [10712.762861] Lustre: server umount lustre-MDT0000 complete [10716.146197] LustreError: 347579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10716.168801] LustreError: 347579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10716.810253] LustreError: 346833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604290 with bad export cookie 15123550214491528745 [10716.814666] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10716.821097] LustreError: 346833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10717.211798] Lustre: server umount lustre-MDT0001 complete [10730.488747] Lustre: server umount lustre-OST0000 complete [10746.060092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10751.548751] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10758.976257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10759.368720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10759.375285] Lustre: Skipped 1 previous similar message [10763.563480] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10770.735268] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10776.042664] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10782.608585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10789.693783] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10792.181677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10797.369976] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10802.660292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10802.663653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10802.665277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10802.665284] Lustre: Skipped 2 previous similar messages [10802.675482] Lustre: Skipped 2 previous similar messages [10808.225226] Lustre: server umount lustre-OST0000 complete [10812.902935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10812.916927] Lustre: Skipped 4 previous similar messages [10818.033851] LustreError: 353194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10818.058085] LustreError: 353194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10818.122288] Lustre: server umount lustre-MDT0000 complete [10822.071659] LustreError: 353179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604395 with bad export cookie 15123550214491529494 [10822.074347] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10822.087408] LustreError: 353179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10822.525648] Lustre: server umount lustre-MDT0001 complete [10833.420898] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [10836.027986] Key type lgssc unregistered [10836.384463] LNet: 356731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10836.399060] LNetError: 356731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10836.427135] LNet: Removed LNI 192.168.204.145@tcp [10837.250980] Key type .llcrypt unregistered [10837.255555] Key type ._llcrypt unregistered [10864.237831] Key type ._llcrypt registered [10864.240751] Key type .llcrypt registered [10864.376343] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [10865.789609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10865.867679] alg: No test for adler32 (adler32-zlib) [10867.092768] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [10867.363568] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [10869.136185] Key type lgssc registered [10870.275771] Lustre: Echo OBD driver; http://www.lustre.org/ [10878.886562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10878.924565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10880.486371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10884.844816] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10890.723089] LustreError: 358244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10890.747758] LustreError: 358244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10893.423906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10893.717345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10898.383813] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10906.381705] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10912.001387] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10918.700249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10918.989995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10925.340789] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10926.136548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10933.226313] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10948.580689] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10948.586283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10948.604642] Lustre: Skipped 2 previous similar messages [10948.617666] Lustre: Skipped 2 previous similar messages [10950.304524] Lustre: server umount lustre-MDT0000 complete [10953.704152] LustreError: 358240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10953.734501] LustreError: 358240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10954.077163] LustreError: 358223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604527 with bad export cookie 6467347963973315513 [10954.087076] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10954.089339] LustreError: 358223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10954.592956] Lustre: server umount lustre-MDT0001 complete [10968.902428] Lustre: server umount lustre-OST0000 complete [10982.757775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10983.096421] LustreError: 361972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10983.164339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10988.516951] LustreError: 361973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10988.740210] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10992.614978] LustreError: 361972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10997.748756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10998.099587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11002.851222] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11010.558528] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11015.623239] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11022.352449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11022.781218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11027.892859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11028.741903] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11036.562683] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11053.545623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11053.556719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11053.567500] Lustre: Skipped 2 previous similar messages [11053.569728] Lustre: Skipped 1 previous similar message [11056.892656] Lustre: server umount lustre-MDT0000 complete [11058.666519] LustreError: 363533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11058.684564] LustreError: 363533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11060.873391] LustreError: 361953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604634 with bad export cookie 6467347963973316626 [11060.895857] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11061.134629] Lustre: server umount lustre-MDT0001 complete [11075.403789] Lustre: server umount lustre-OST0000 complete [11092.374834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11092.854720] LustreError: 365885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11092.965877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11097.843359] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11106.682605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11111.458334] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11118.343529] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11123.585894] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11132.669884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11133.193692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11133.199499] Lustre: Skipped 1 previous similar message [11137.275278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11140.138531] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11147.432135] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11152.874973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11152.890956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11152.916382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11152.921525] Lustre: Skipped 1 previous similar message [11156.963122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11156.967409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11156.968312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11156.988175] Lustre: Skipped 1 previous similar message [11159.034772] Lustre: server umount lustre-OST0000 complete [11167.203726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11167.206792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11167.216647] Lustre: Skipped 1 previous similar message [11169.894484] Lustre: server umount lustre-MDT0000 complete [11172.322754] LustreError: 365879:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11172.368252] LustreError: 365879:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11173.234526] LustreError: 365867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604747 with bad export cookie 6467347963973317648 [11173.241737] LustreError: 365867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11173.244799] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11173.518629] Lustre: server umount lustre-MDT0001 complete [11182.995663] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [11185.584929] Key type lgssc unregistered [11185.923803] LNet: 369417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11185.932352] LNetError: 369417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11186.982836] LNet: Removed LNI 192.168.204.145@tcp [11187.945832] Key type .llcrypt unregistered [11187.950251] Key type ._llcrypt unregistered [11214.664819] Key type ._llcrypt registered [11214.673255] Key type .llcrypt registered [11214.822944] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [11216.617428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11216.655331] alg: No test for adler32 (adler32-zlib) [11217.684323] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [11217.878725] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [11219.529767] Key type lgssc registered [11221.144145] Lustre: Echo OBD driver; http://www.lustre.org/ [11230.469106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11230.481426] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11230.487458] LustreError: 370853:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11230.502740] LustreError: 370853:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11230.509849] LustreError: 370853:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11230.524460] LustreError: 370853:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11230.533994] LustreError: 370853:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11238.291510] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11238.295271] LustreError: 371251:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11238.313634] LustreError: 371251:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11238.325380] LustreError: 371251:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11238.331262] LustreError: 371251:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11238.336497] LustreError: 371251:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11259.146329] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [11262.542190] Key type lgssc unregistered [11262.825843] LNet: 372253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11262.831876] LNetError: 372253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11262.845475] LNet: Removed LNI 192.168.204.145@tcp [11263.642062] Key type .llcrypt unregistered [11263.646724] Key type ._llcrypt unregistered [11283.531720] Key type ._llcrypt registered [11283.534229] Key type .llcrypt registered [11283.632770] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [11300.667746] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [11301.991335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11302.093804] alg: No test for adler32 (adler32-zlib) [11303.416771] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [11303.856953] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [11305.617192] Key type lgssc registered [11306.909960] Lustre: Echo OBD driver; http://www.lustre.org/ [11342.481898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11342.531817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11343.814908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11343.851369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11343.919280] Lustre: lustre-MDT0000: new disk, initializing [11343.996777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11344.023411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11349.061922] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11359.808207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11360.012242] Lustre: 375406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11360.088697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11360.093387] Lustre: Skipped 1 previous similar message [11360.190446] Lustre: lustre-MDT0001: new disk, initializing [11360.388300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11360.450147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11360.478869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11365.827780] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11376.953266] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11383.570588] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11391.817276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11392.125230] Lustre: lustre-OST0000: new disk, initializing [11392.128660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11392.145239] Lustre: 377377:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11392.285168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11398.747330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11398.754284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11398.940533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11399.704476] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11410.225133] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11415.985902] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11416.196509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11421.717462] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11421.990492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11424.738104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11424.741082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11424.752060] LustreError: Skipped 1 previous similar message [11424.753319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11424.788546] Lustre: Skipped 1 previous similar message [11429.866430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11429.871835] Lustre: Skipped 1 previous similar message [11430.317144] Lustre: server umount lustre-OST0000 complete [11434.986361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11434.993700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11434.997026] Lustre: Skipped 1 previous similar message [11435.007962] Lustre: Skipped 2 previous similar messages [11440.100269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11440.123921] Lustre: Skipped 1 previous similar message [11440.954817] Lustre: server umount lustre-MDT0000 complete [11445.258574] LustreError: 375397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605019 with bad export cookie 3964041806859545197 [11445.269309] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11445.274670] LustreError: 375397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11445.673889] Lustre: server umount lustre-MDT0001 complete [11454.408957] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11456.233961] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 06:43:48 (1788605028) [11463.532377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11464.128079] LustreError: 379837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11464.146398] LustreError: 379837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11464.223340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11469.289970] LustreError: 379838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11469.435039] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11473.382052] LustreError: 379837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11478.498038] LustreError: 379838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11479.985600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11480.406069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11486.095368] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11494.262529] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11500.254212] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11507.360068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11507.889259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11514.735945] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11522.626553] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11530.253061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11530.432824] Lustre: lustre-OST0001: new disk, initializing [11530.439333] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11530.448511] Lustre: 382674:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11530.548977] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11538.446069] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11538.582087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11538.592137] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11538.745801] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11549.846516] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11554.277284] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11554.286466] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11554.294178] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11554.306230] Lustre: Skipped 1 previous similar message [11554.327624] Lustre: Skipped 1 previous similar message [11560.128354] Lustre: server umount lustre-OST0001 complete [11561.738364] LustreError: 381661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11574.757128] LustreError: 383041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11574.779244] LustreError: 383041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11579.881566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11579.909450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11579.913813] Lustre: Skipped 4 previous similar messages [11583.824367] Lustre: server umount lustre-OST0000 complete [11588.065524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11588.077312] LustreError: Skipped 1 previous similar message [11588.083474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11588.094439] Lustre: Skipped 1 previous similar message [11594.034758] Lustre: server umount lustre-MDT0000 complete [11595.238970] LustreError: 379837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11595.282758] LustreError: 379837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11598.468286] LustreError: 379819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605172 with bad export cookie 3964041806859546079 [11598.474930] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11605.025894] Lustre: server umount lustre-MDT0001 complete [11617.630302] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [11621.353213] Key type lgssc unregistered [11621.686798] LNet: 384663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11621.702856] LNetError: 384663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11621.733645] LNet: Removed LNI 192.168.204.145@tcp [11622.835990] Key type .llcrypt unregistered [11622.839978] Key type ._llcrypt unregistered [11680.163872] Key type ._llcrypt registered [11680.171737] Key type .llcrypt registered [11680.291974] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [11681.637821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11681.676548] alg: No test for adler32 (adler32-zlib) [11682.835533] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [11683.113594] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [11684.800219] Key type lgssc registered [11686.045315] Lustre: Echo OBD driver; http://www.lustre.org/ [11695.445686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11695.474033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11696.758067] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11696.786315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11696.965570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11701.840415] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11710.902596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11711.082765] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11711.131811] Lustre: 387459:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11711.542706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11716.653581] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11726.249753] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11733.127389] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11737.059456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11737.064287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11737.078490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11739.618482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11739.619026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11742.054137] Lustre: server umount lustre-MDT0000 complete [11745.648286] LustreError: 387452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605319 with bad export cookie 2754810673222169248 [11745.655657] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11745.667205] LustreError: 387452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11746.121980] Lustre: server umount lustre-MDT0001 complete [11756.347533] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 06:48:48 (1788605328) [11763.804604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11764.265590] LustreError: 389977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11764.286892] LustreError: 389977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11764.345741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11768.355708] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11769.835631] LustreError: 389978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11774.954361] LustreError: 389977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11777.784592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11778.185474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11783.657804] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11792.487775] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11798.762818] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11806.965946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11807.173797] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11807.490415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11814.833786] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11824.629754] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11833.472705] Lustre: Permanently deactivating lustre-OST0000 [11833.481388] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11842.600383] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11844.847912] LustreError: 392965:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11844.864931] LustreError: 392965:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11844.884434] LustreError: 392965:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11848.920620] LustreError: 393014:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11848.939261] LustreError: 393014:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11853.950797] Lustre: server umount lustre-OST0000 complete [11856.816102] LustreError: 393263:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11856.820378] LustreError: 393263:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11859.443530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11859.464729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11859.473875] Lustre: Skipped 2 previous similar messages [11864.545343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11864.567978] Lustre: Skipped 1 previous similar message [11869.665588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11869.678996] Lustre: Skipped 1 previous similar message [11873.254241] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11873.592605] Lustre: server umount lustre-MDT0000 complete [11874.789115] LustreError: 389977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11874.798674] LustreError: 389977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11879.003443] LustreError: 389958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605452 with bad export cookie 2754810673222169836 [11879.026419] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11879.034630] LustreError: 389958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11888.861178] Lustre: server umount lustre-MDT0001 complete [11898.247818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11903.461867] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11906.095312] LustreError: 394422:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11913.007351] Lustre: 394761:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.145@tcp [11915.853028] Lustre: server umount MGS complete [11930.427919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11930.977990] LustreError: 395485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11931.000333] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11931.011930] Lustre: Skipped 1 previous similar message [11931.077877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11937.145915] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11941.346020] LustreError: 395486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11941.375087] LustreError: 395486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11947.757846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11948.021261] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11948.127077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11953.074326] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11961.839898] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11969.193169] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11971.648488] Lustre: Permanently reactivating lustre-OST0000 [11971.671975] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11971.683571] Lustre: Skipped 2 previous similar messages [11999.202597] LustreError: 397239:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12007.867423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12008.271648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12014.784150] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12022.556178] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12028.898805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12028.919317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12028.943920] Lustre: Skipped 1 previous similar message [12028.954531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12029.929032] Lustre: server umount lustre-OST0000 complete [12035.043844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12035.046603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12044.464220] Lustre: server umount lustre-MDT0000 complete [12048.752803] LustreError: 397062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605622 with bad export cookie 2754810673222170886 [12048.761964] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12048.766459] LustreError: 397062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12049.111734] Lustre: server umount lustre-MDT0001 complete [12061.392373] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12065.173921] Key type lgssc unregistered [12065.485751] LNet: 399854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12065.497340] LNetError: 399854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12065.523504] LNet: Removed LNI 192.168.204.145@tcp [12066.518510] Key type .llcrypt unregistered [12066.521235] Key type ._llcrypt unregistered [12087.362480] Key type ._llcrypt registered [12087.367637] Key type .llcrypt registered [12087.506948] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [12105.223040] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12106.288511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12106.352572] alg: No test for adler32 (adler32-zlib) [12107.492986] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [12107.869561] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12109.568231] Key type lgssc registered [12110.884510] Lustre: Echo OBD driver; http://www.lustre.org/ [12147.161541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12147.176283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12148.438644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12148.507653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12148.625952] Lustre: lustre-MDT0000: new disk, initializing [12148.678302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12148.694379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12153.328455] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12164.473573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12164.600397] Lustre: 403009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12164.682520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12164.689628] Lustre: Skipped 1 previous similar message [12164.800733] Lustre: lustre-MDT0001: new disk, initializing [12164.934996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12164.993164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12164.999760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12169.579162] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12179.251293] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12184.818838] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12190.886296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12191.278798] Lustre: lustre-OST0000: new disk, initializing [12191.288613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12191.304082] Lustre: 404977:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12191.464723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12197.440979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12197.453682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12197.547098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12198.459581] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12209.198819] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12214.758187] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12214.981752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12219.869109] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12220.085984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12223.460799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12223.464822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12223.475113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12223.505763] LustreError: Skipped 1 previous similar message [12228.581154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12228.588419] Lustre: Skipped 2 previous similar messages [12233.701809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12233.708358] Lustre: Skipped 1 previous similar message [12236.256300] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12236.606451] Lustre: server umount lustre-OST0000 complete [12241.899242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12241.906816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12241.921293] Lustre: Skipped 1 previous similar message [12241.926836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12243.938635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12247.110965] Lustre: server umount lustre-MDT0000 complete [12250.588271] LustreError: 403001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605824 with bad export cookie 6651372378505983300 [12250.598268] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12250.600106] LustreError: 403001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12250.950612] Lustre: server umount lustre-MDT0001 complete [12260.078703] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12261.721544] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 06:57:13 (1788605833) [12269.069780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12269.526575] LustreError: 407437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12269.549274] LustreError: 407437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12269.620332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12274.665300] LustreError: 407438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12275.230095] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12279.780157] LustreError: 407437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12284.699470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12284.896994] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12284.905071] Lustre: Skipped 2 previous similar messages [12284.993899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12288.613370] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12296.593184] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12304.051775] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12312.665095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12313.184877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12319.669158] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12326.962127] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12328.968390] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12333.436110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12345.313089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12345.317324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12345.318181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12345.333502] Lustre: Skipped 1 previous similar message [12347.512830] Lustre: server umount lustre-OST0000 complete [12355.555584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12355.586579] Lustre: Skipped 1 previous similar message [12359.211780] Lustre: server umount lustre-MDT0000 complete [12360.673175] LustreError: 407438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12360.686042] LustreError: 407438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12362.899885] LustreError: 407417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605936 with bad export cookie 6651372378505984182 [12362.909750] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12362.916265] LustreError: 407417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12363.378186] Lustre: server umount lustre-MDT0001 complete [12375.315512] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12378.327541] Key type lgssc unregistered [12378.648849] LNet: 411165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12378.663058] LNetError: 411165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12378.708267] LNet: Removed LNI 192.168.204.145@tcp [12379.662139] Key type .llcrypt unregistered [12379.665433] Key type ._llcrypt unregistered [12404.857864] Key type ._llcrypt registered [12404.860507] Key type .llcrypt registered [12404.984504] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12405.300238] Key type .llcrypt unregistered [12405.303454] Key type ._llcrypt unregistered [12423.962571] Key type ._llcrypt registered [12423.964439] Key type .llcrypt registered [12424.076751] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12424.886606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12425.018894] alg: No test for adler32 (adler32-zlib) [12426.191482] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [12426.494769] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12428.193453] Key type lgssc registered [12429.520609] Lustre: Echo OBD driver; http://www.lustre.org/ [12437.043371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12437.076551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12438.562216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12443.352345] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12448.739978] LustreError: 413376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12448.756226] LustreError: 413376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12450.668589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12450.927933] LustreError: 413375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12450.959065] LustreError: 413375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12450.960918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12456.418219] LustreError: 413372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12456.440103] LustreError: 413372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12456.635414] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12461.549418] LustreError: 413371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12461.570348] LustreError: 413371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12463.611792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12463.758399] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12468.708585] LustreError: 413376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12468.738047] LustreError: 413376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12470.956660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12471.335207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12476.018369] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12486.629420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12486.640090] Lustre: Skipped 1 previous similar message [12486.642659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12486.643081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12486.673246] Lustre: Skipped 1 previous similar message [12487.109555] Lustre: server umount lustre-OST0000 complete [12491.749252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12491.755340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12491.760722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12491.766559] Lustre: Skipped 1 previous similar message [12496.870817] LustreError: 414436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12496.883872] LustreError: 414436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12496.993733] Lustre: server umount lustre-MDT0000 complete [12500.940191] LustreError: 413355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606074 with bad export cookie 10028501872228209936 [12500.943231] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12500.952758] LustreError: 413355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12501.294509] Lustre: server umount lustre-MDT0001 complete [12513.905491] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12518.302811] Key type lgssc unregistered [12518.608513] LNet: 416533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12518.629520] LNetError: 416533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12518.656299] LNet: Removed LNI 192.168.204.145@tcp [12519.687179] Key type .llcrypt unregistered [12519.694262] Key type ._llcrypt unregistered [12542.130285] Key type ._llcrypt registered [12542.134798] Key type .llcrypt registered [12542.242425] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12543.127028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12543.264558] alg: No test for adler32 (adler32-zlib) [12544.392319] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [12544.621919] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12546.316720] Key type lgssc registered [12547.677596] Lustre: Echo OBD driver; http://www.lustre.org/ [12555.833199] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12555.870993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12571.425060] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12576.544411] LustreError: 417921:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [12603.232198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12610.605495] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12617.769146] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12624.549617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12625.147247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12625.225289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12630.117822] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12630.509185] LustreError: 417947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12630.532033] LustreError: 417947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12635.619542] LustreError: 417946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12637.699770] LustreError: 417947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12637.725265] LustreError: 417947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12639.951840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12640.438325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12645.570226] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12652.880192] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12658.551714] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12665.826159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12665.834253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12665.853964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12666.346270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12666.347265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12666.372941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12666.382314] Lustre: Skipped 1 previous similar message [12667.542695] Lustre: server umount lustre-OST0000 complete [12670.949630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12670.967360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12670.989666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12676.577133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12676.584438] Lustre: Skipped 3 previous similar messages [12676.815875] Lustre: server umount lustre-MDT0000 complete [12680.174766] LustreError: 417928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606254 with bad export cookie 3136053673123418293 [12680.176769] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12680.187857] LustreError: 417928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12680.528575] Lustre: server umount lustre-MDT0001 complete [12690.849948] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12694.008373] Key type lgssc unregistered [12694.463861] LNet: 421488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12694.476812] LNetError: 421488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12695.535587] LNet: Removed LNI 192.168.204.145@tcp [12696.391819] Key type .llcrypt unregistered [12696.395055] Key type ._llcrypt unregistered [12719.951905] Key type ._llcrypt registered [12719.953600] Key type .llcrypt registered [12720.071724] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12721.117232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12721.294978] alg: No test for adler32 (adler32-zlib) [12722.551640] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [12723.000199] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12724.792198] Key type lgssc registered [12726.290134] Lustre: Echo OBD driver; http://www.lustre.org/ [12735.147876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12735.189930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12736.861839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12741.330711] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12746.214985] LustreError: 422904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12746.240522] LustreError: 422904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12750.277602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12750.947429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12755.874935] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12762.498553] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12768.265746] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12774.940640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12775.325673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12781.490531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12781.981504] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12789.314297] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12796.215793] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [12796.900300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12796.907722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12796.914532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12796.926169] Lustre: Skipped 1 previous similar message [12799.571337] Lustre: server umount lustre-MDT0000 complete [12801.289266] LustreError: 422899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12802.163348] LustreError: 424745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.45@tcp arrived at 1788606376 with bad export cookie 1434767430391766814 [12802.179671] LustreError: 424745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12804.255681] LustreError: 422883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606378 with bad export cookie 1434767430391765918 [12804.268320] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12804.273419] LustreError: 422883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12807.155086] LustreError: 422899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12807.160258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12807.170141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12807.188775] LustreError: 422899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12807.217618] Lustre: Skipped 1 previous similar message [12807.230667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12807.237513] Lustre: Skipped 2 previous similar messages [12809.187524] LustreError: 422903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12809.210273] LustreError: 422903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12810.442611] Lustre: server umount lustre-OST0000 complete [12814.305654] LustreError: 422899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12814.320248] LustreError: 422899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12816.288270] Lustre: server umount lustre-MDT0001 complete [12827.313515] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12830.094231] Key type lgssc unregistered [12830.414659] LNet: 426585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12830.426796] LNetError: 426585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12830.455725] LNet: Removed LNI 192.168.204.145@tcp [12831.257321] Key type .llcrypt unregistered [12831.259405] Key type ._llcrypt unregistered [12854.638673] Key type ._llcrypt registered [12854.641465] Key type .llcrypt registered [12854.730035] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12855.482630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12855.517055] alg: No test for adler32 (adler32-zlib) [12856.645995] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [12857.036505] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12858.672172] Key type lgssc registered [12859.897908] Lustre: Echo OBD driver; http://www.lustre.org/ [12867.429767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12867.444428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12868.891224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12872.807803] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12878.307244] LustreError: 427997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12878.317727] LustreError: 427997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12881.025238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12881.298254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12885.375861] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12892.610880] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12897.105336] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12903.287848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12903.613239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12910.411755] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12913.855330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12918.044678] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12922.360797] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12922.375783] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12922.396276] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12924.232535] Lustre: server umount lustre-MDT0001 complete [12924.388246] LustreError: 427996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12924.400338] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12929.507325] LustreError: 429236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12929.524994] LustreError: 429236:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12934.625419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12934.626575] LustreError: 427997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12934.634973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12934.636832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12935.980726] Lustre: server umount lustre-OST0000 complete [12939.951780] Lustre: server umount lustre-MDT0000 complete [12955.725488] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12959.489389] Key type lgssc unregistered [12959.780782] LNet: 431677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12959.788450] LNetError: 431677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12959.802782] LNet: Removed LNI 192.168.204.145@tcp [12960.675216] Key type .llcrypt unregistered [12960.679949] Key type ._llcrypt unregistered [12989.123716] Key type ._llcrypt registered [12989.125261] Key type .llcrypt registered [12989.271166] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [13005.451912] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [13006.699140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13006.768097] alg: No test for adler32 (adler32-zlib) [13007.800524] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [13008.093890] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [13009.832189] Key type lgssc registered [13011.288231] Lustre: Echo OBD driver; http://www.lustre.org/ [13053.190596] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [13068.508211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13068.527683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13069.827173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13069.876395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13070.027323] Lustre: lustre-MDT0000: new disk, initializing [13070.149451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13070.169722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13074.810547] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13091.238459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13091.621437] Lustre: lustre-OST0000: new disk, initializing [13091.626438] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13091.630547] Lustre: Skipped 1 previous similar message [13091.641373] Lustre: 436710:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13091.738963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13098.612149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13098.617970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13098.656559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13099.089182] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13109.594697] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13124.542960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13124.713683] Lustre: 435689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13124.781613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13124.884338] Lustre: lustre-MDT0001: new disk, initializing [13125.023268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13125.108577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13125.114886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13129.471498] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13140.412849] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13140.765782] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13145.783267] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13146.109753] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13151.041715] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [13152.737602] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13152.750329] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13152.760522] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13152.802726] Lustre: Skipped 1 previous similar message [13156.105313] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [13156.115818] Lustre: Skipped 1 previous similar message [13157.379676] Lustre: server umount lustre-MDT0001 complete [13162.997724] LustreError: 435703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13163.026037] LustreError: 435703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13168.102465] LustreError: 435699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13168.129239] LustreError: 435699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13173.227478] LustreError: 435697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13173.267406] LustreError: 435697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13175.778661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13175.790415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13175.814299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13178.338464] LustreError: 435702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13181.456552] Lustre: server umount lustre-OST0000 complete [13183.459869] LustreError: 438411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13192.743812] Lustre: server umount lustre-MDT0000 complete [13208.684430] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [13211.798392] Key type lgssc unregistered [13212.190687] LNet: 440460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13212.204500] LNetError: 440460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13212.236841] LNet: Removed LNI 192.168.204.145@tcp [13213.318558] Key type .llcrypt unregistered [13213.337409] Key type ._llcrypt unregistered [13257.004744] Key type ._llcrypt registered [13257.008059] Key type .llcrypt registered [13257.159555] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [13257.496377] Key type .llcrypt unregistered [13257.498674] Key type ._llcrypt unregistered [13275.809506] Key type ._llcrypt registered [13275.812619] Key type .llcrypt registered [13275.920464] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [13277.429141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13277.476358] alg: No test for adler32 (adler32-zlib) [13278.674333] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [13278.897601] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [13280.624560] Key type lgssc registered [13282.131594] Lustre: Echo OBD driver; http://www.lustre.org/ [13310.208454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13310.242979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13311.565644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13311.619471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13311.716335] Lustre: lustre-MDT0000: new disk, initializing [13311.825193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13311.852814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13316.521631] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13327.844482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13328.026952] Lustre: 443829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13328.099228] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13328.108315] Lustre: Skipped 1 previous similar message [13328.204060] Lustre: lustre-MDT0001: new disk, initializing [13328.337797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13328.402419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13328.417989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13333.211183] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13344.335481] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13350.952866] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13358.684637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13358.965885] Lustre: lustre-OST0000: new disk, initializing [13358.973019] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13358.984651] Lustre: 445796:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13359.140260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13365.267119] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13365.311600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13365.320167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13365.448564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13375.003901] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13384.677269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13384.688778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13384.713755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13386.213522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13386.214548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13386.231431] Lustre: Skipped 1 previous similar message [13386.245095] Lustre: Skipped 2 previous similar messages [13387.683689] Lustre: server umount lustre-MDT0000 complete [13391.700166] LustreError: 443821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606965 with bad export cookie 8734520878867094278 [13391.720204] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13392.278215] Lustre: server umount lustre-MDT0001 complete [13406.441396] Lustre: server umount lustre-OST0000 complete [13417.673260] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 07:16:29 (1788606989) [13425.333370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13425.827917] LustreError: 447830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13425.858967] LustreError: 447830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13425.929147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13429.965994] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13431.267358] LustreError: 447831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13435.365861] LustreError: 447830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13438.420320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13438.795972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13443.580291] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13451.074890] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13456.707460] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13476.978023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13477.280602] Lustre: Found index 0 for lustre-OST0000, updating log [13477.509726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13483.813785] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13491.979645] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13497.024447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13501.412216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13501.421086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13501.440362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13503.327189] Lustre: server umount lustre-OST0000 complete [13520.866474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13520.873580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13520.890868] Lustre: Skipped 1 previous similar message [13520.899246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13520.905919] Lustre: Skipped 3 previous similar messages [13525.457456] Lustre: server umount lustre-MDT0000 complete [13529.061510] LustreError: 447831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13529.112695] LustreError: 447831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13529.823904] LustreError: 447812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788607103 with bad export cookie 8734520878867095664 [13529.830747] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13529.834298] LustreError: 447812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [13530.139339] Lustre: server umount lustre-MDT0001 complete [13539.138742] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 07:18:31 (1788607111) [13541.088161] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp [14159.963089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14160.533600] LustreError: 454239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14160.598565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14165.285520] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14165.987664] LustreError: 454240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14171.113134] LustreError: 454239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14174.857792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14175.363372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14180.102305] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14187.367559] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14192.055230] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14210.933394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14211.213662] Lustre: Found index 0 for lustre-OST0000, updating log [14211.531200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14217.682913] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14220.000119] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp [14224.319126] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14226.278777] Lustre: server umount lustre-OST0000 complete [14230.755214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14230.759332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14230.766370] Lustre: Skipped 2 previous similar messages [14230.777030] Lustre: Skipped 3 previous similar messages [14231.784994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14231.791659] Lustre: Skipped 1 previous similar message [14232.808422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14232.813924] Lustre: Skipped 1 previous similar message [14234.851468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14234.856912] Lustre: Skipped 3 previous similar messages [14238.951174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14238.957564] Lustre: Skipped 7 previous similar messages [14244.320907] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14244.619199] Lustre: server umount lustre-MDT0000 complete [14245.102714] LustreError: 456703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14245.122708] LustreError: 456703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14247.888522] LustreError: 454221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788607821 with bad export cookie 8734520878867096749 [14247.892870] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14247.897430] LustreError: 454221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [14248.221662] Lustre: server umount lustre-MDT0001 complete [14278.630528] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [14283.490079] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp [14288.286127] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [14324.169405] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 07:31:36 (1788607896) [14341.048594] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [14343.409966] Key type lgssc unregistered [14343.722278] LNet: 460849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14343.730577] LNetError: 460849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14472.813065] LNet: Removed LNI 192.168.204.145@tcp [14473.720796] Key type .llcrypt unregistered [14473.724293] Key type ._llcrypt unregistered [14492.926883] Key type ._llcrypt registered [14492.931880] Key type .llcrypt registered [14493.093086] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [14494.337456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14494.627241] alg: No test for adler32 (adler32-zlib) [14495.603931] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [14495.815551] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [14497.520178] Key type lgssc registered [14498.985965] Lustre: Echo OBD driver; http://www.lustre.org/ [14502.710469] LNet: Added LNI 192.168.204.145@tcp100 [8/256/0/180] [14503.449965] LNet: Added LNI 192.168.204.145@tcp101 [8/256/0/180] [14504.288965] LNet: Added LNI 192.168.204.145@tcp102 [8/256/0/180] [14505.075206] LNet: Added LNI 192.168.204.145@tcp103 [8/256/0/180] [14505.922726] LNet: Added LNI 192.168.204.145@tcp104 [8/256/0/180] [14506.565576] LNet: Added LNI 192.168.204.145@tcp105 [8/256/0/180] [14507.327792] LNet: Added LNI 192.168.204.145@tcp106 [8/256/0/180] [14508.048993] LNet: Added LNI 192.168.204.145@tcp107 [8/256/0/180] [14508.829645] LNet: Added LNI 192.168.204.145@tcp108 [8/256/0/180] [14509.525307] LNet: Added LNI 192.168.204.145@tcp109 [8/256/0/180] [14510.099762] LNet: Added LNI 192.168.204.145@tcp110 [8/256/0/180] [14510.887250] LNet: Added LNI 192.168.204.145@tcp111 [8/256/0/180] [14511.557111] LNet: Added LNI 192.168.204.145@tcp112 [8/256/0/180] [14512.178854] LNet: Added LNI 192.168.204.145@tcp113 [8/256/0/180] [14512.789366] LNet: Added LNI 192.168.204.145@tcp114 [8/256/0/180] [14513.468496] LNet: Added LNI 192.168.204.145@tcp115 [8/256/0/180] [14514.109597] LNet: Added LNI 192.168.204.145@tcp116 [8/256/0/180] [14514.784732] LNet: Added LNI 192.168.204.145@tcp117 [8/256/0/180] [14515.687588] LNet: Added LNI 192.168.204.145@tcp118 [8/256/0/180] [14516.627918] LNet: Added LNI 192.168.204.145@tcp119 [8/256/0/180] [14517.561425] LNet: Added LNI 192.168.204.145@tcp120 [8/256/0/180] [14518.555679] LNet: Added LNI 192.168.204.145@tcp121 [8/256/0/180] [14519.556193] LNet: Added LNI 192.168.204.145@tcp122 [8/256/0/180] [14520.582772] LNet: Added LNI 192.168.204.145@tcp123 [8/256/0/180] [14521.618822] LNet: Added LNI 192.168.204.145@tcp124 [8/256/0/180] [14522.402044] LNet: Added LNI 192.168.204.145@tcp125 [8/256/0/180] [14523.211910] LNet: Added LNI 192.168.204.145@tcp126 [8/256/0/180] [14524.056312] LNet: Added LNI 192.168.204.145@tcp127 [8/256/0/180] [14524.937975] LNet: Added LNI 192.168.204.145@tcp128 [8/256/0/180] [14525.854943] LNet: Added LNI 192.168.204.145@tcp129 [8/256/0/180] [14526.736554] LNet: Added LNI 192.168.204.145@tcp130 [8/256/0/180] [14527.611561] LNet: Added LNI 192.168.204.145@tcp131 [8/256/0/180] [14528.479362] LNet: Added LNI 192.168.204.145@tcp132 [8/256/0/180] [14529.366615] LNet: Added LNI 192.168.204.145@tcp133 [8/256/0/180] [14530.176723] LNet: Added LNI 192.168.204.145@tcp134 [8/256/0/180] [14530.976136] LNet: Added LNI 192.168.204.145@tcp135 [8/256/0/180] [14537.831616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14537.860774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14539.024594] Lustre: 463880:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [14539.140666] LNetError: 463901:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14539.176512] LNetError: Protocol error connecting to 192.168.204.145@tcp123 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14539.204775] LNetError: Skipped 4 previous similar messages [14539.404268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14539.459674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14539.544373] Lustre: lustre-MDT0000: new disk, initializing [14539.620948] Lustre: 463936:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [14539.628552] Lustre: 463936:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14539.645786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14539.661174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14539.872338] LNet: 4 local NIs in recovery (showing 4): 192.168.204.145@tcp127, 192.168.204.145@tcp128, 192.168.204.145@tcp129, 192.168.204.145@tcp123 [14540.257034] LNetError: 461339:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14540.275103] LNetError: 461339:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14540.281450] LNetError: Protocol error connecting to 192.168.204.145@tcp104 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14540.290276] LNetError: Skipped 3 previous similar messages [14542.945694] LNetError: 463913:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14542.969265] LNetError: 463913:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14542.987582] LNetError: Protocol error connecting to 192.168.204.145@tcp128 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14542.996042] LNetError: Skipped 4 previous similar messages [14544.232723] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14552.959372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14553.073404] Lustre: 463898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14553.125571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14553.128434] Lustre: Skipped 1 previous similar message [14553.189255] Lustre: lustre-MDT0001: new disk, initializing [14553.224763] Lustre: 464699:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [14553.232970] Lustre: 464699:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14553.320951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14553.347347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14553.358492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14554.211179] LNetError: 461337:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14554.220894] LNetError: Protocol error connecting to 192.168.204.145@tcp128 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14556.428696] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14564.283486] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14568.880883] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14570.595142] LNetError: 463905:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14570.611523] LNetError: Protocol error connecting to 192.168.204.145@tcp128 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14574.519523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14574.787800] Lustre: lustre-OST0000: new disk, initializing [14574.793038] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14574.799919] Lustre: 465870:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14574.872296] Lustre: 465875:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [14574.877700] Lustre: 465875:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14574.928781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14579.750146] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14579.812321] LNetError: 463901:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14579.828279] LNetError: 463901:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14579.834290] LNetError: Protocol error connecting to 192.168.204.145@tcp122 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14579.840512] LNetError: Skipped 2 previous similar messages [14581.766522] Lustre: 466326:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [14581.777571] Lustre: 466326:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14581.829814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14581.841481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14581.923572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14588.060603] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14597.604094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14597.610131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14597.623484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14597.634602] Lustre: Skipped 1 previous similar message [14599.265729] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp128 [14599.266462] LNetError: 463902:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14599.289768] LNetError: 463902:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14599.303675] LNetError: Protocol error connecting to 192.168.204.145@tcp126 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14599.317341] LNetError: Skipped 7 previous similar messages [14602.725199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14602.729903] Lustre: Skipped 2 previous similar messages [14607.844292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14607.850759] Lustre: Skipped 1 previous similar message [14608.353447] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14608.493883] Lustre: server umount lustre-OST0000 complete [14612.962651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14612.964979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14612.978255] Lustre: Skipped 1 previous similar message [14612.989469] Lustre: Skipped 1 previous similar message [14618.085876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14618.090320] Lustre: Skipped 1 previous similar message [14626.272143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14626.431134] Lustre: server umount lustre-MDT0000 complete [14629.551567] LustreError: 465869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788608203 with bad export cookie 3816164399621901797 [14629.554806] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14629.566951] LustreError: 465869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14630.064814] Lustre: server umount lustre-MDT0001 complete [14632.039558] LNetError: 463902:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14632.061332] LNetError: 463902:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14632.070628] LNetError: Protocol error connecting to 192.168.204.145@tcp104 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14632.085293] LNetError: Skipped 9 previous similar messages [14641.262765] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [14644.016584] Key type lgssc unregistered [14644.463311] LNet: 467784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14644.479231] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14644.489693] LNet: Removed LNI 192.168.204.145@tcp [14645.536778] LNet: Removed LNI 192.168.204.145@tcp100 [14645.543782] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14645.550918] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14645.578353] LNet: Removed LNI 192.168.204.145@tcp101 [14646.624188] LNet: Removed LNI 192.168.204.145@tcp102 [14646.626426] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14646.637210] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14646.653956] LNet: Removed LNI 192.168.204.145@tcp103 [14647.712762] LNet: Removed LNI 192.168.204.145@tcp104 [14648.736357] LNet: Removed LNI 192.168.204.145@tcp105 [14648.741359] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14648.761108] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14648.776392] LNet: Removed LNI 192.168.204.145@tcp106 [14649.824895] LNet: Removed LNI 192.168.204.145@tcp107 [14650.848589] LNet: Removed LNI 192.168.204.145@tcp108 [14651.872721] LNet: Removed LNI 192.168.204.145@tcp109 [14652.896230] LNet: Removed LNI 192.168.204.145@tcp110 [14652.899286] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14652.909488] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14652.916288] LNet: Removed LNI 192.168.204.145@tcp111 [14653.984208] LNet: Removed LNI 192.168.204.145@tcp112 [14655.008224] LNet: Removed LNI 192.168.204.145@tcp113 [14656.032194] LNet: Removed LNI 192.168.204.145@tcp114 [14657.056947] LNet: Removed LNI 192.168.204.145@tcp115 [14658.080193] LNet: Removed LNI 192.168.204.145@tcp116 [14659.104148] LNet: Removed LNI 192.168.204.145@tcp117 [14660.128263] LNet: Removed LNI 192.168.204.145@tcp118 [14661.155160] LNet: Removed LNI 192.168.204.145@tcp119 [14661.161984] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14661.170074] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14661.180968] LNet: Removed LNI 192.168.204.145@tcp120 [14662.241644] LNet: Removed LNI 192.168.204.145@tcp121 [14663.266421] LNet: Removed LNI 192.168.204.145@tcp122 [14664.288230] LNet: Removed LNI 192.168.204.145@tcp123 [14665.312203] LNet: Removed LNI 192.168.204.145@tcp124 [14666.336196] LNet: Removed LNI 192.168.204.145@tcp125 [14667.360180] LNet: Removed LNI 192.168.204.145@tcp126 [14668.387297] LNet: Removed LNI 192.168.204.145@tcp127 [14668.389378] LNet: Removed LNI 192.168.204.145@tcp128 [14669.408506] LNet: Removed LNI 192.168.204.145@tcp129 [14669.410447] LNet: Removed LNI 192.168.204.145@tcp130 [14669.416687] LNet: Removed LNI 192.168.204.145@tcp131 [14669.419344] LNet: Removed LNI 192.168.204.145@tcp132 [14669.422516] LNet: Removed LNI 192.168.204.145@tcp133 [14669.425523] LNet: Removed LNI 192.168.204.145@tcp134 [14669.434413] LNet: Removed LNI 192.168.204.145@tcp135 [14670.280515] Key type .llcrypt unregistered [14670.284197] Key type ._llcrypt unregistered [14692.032274] Key type ._llcrypt registered [14692.036313] Key type .llcrypt registered [14692.152150] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [14707.511859] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [14708.959917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14709.016535] alg: No test for adler32 (adler32-zlib) [14710.088337] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [14710.417116] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [14712.280223] Key type lgssc registered [14713.902074] Lustre: Echo OBD driver; http://www.lustre.org/ [14751.173249] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:38:42 (1788608322) [14771.944527] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [14775.275086] Key type lgssc unregistered [14775.609609] LNet: 471870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14775.625410] LNetError: 471870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14775.642154] LNet: Removed LNI 192.168.204.145@tcp [14776.526680] Key type .llcrypt unregistered [14776.535344] Key type ._llcrypt unregistered [14794.091654] Key type ._llcrypt registered [14794.095336] Key type .llcrypt registered [14794.259713] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [14795.700145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14795.771078] alg: No test for adler32 (adler32-zlib) [14797.007703] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [14797.317364] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [14799.048625] Key type lgssc registered [14800.265348] Lustre: Echo OBD driver; http://www.lustre.org/ [14811.857731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14811.898801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14813.219248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14813.275889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14813.361862] Lustre: lustre-MDT0000: new disk, initializing [14813.442080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14813.466830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14817.451146] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14828.329845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14828.472538] Lustre: 473079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14828.525367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14828.530848] Lustre: Skipped 1 previous similar message [14828.623886] Lustre: lustre-MDT0001: new disk, initializing [14828.780732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14828.838705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14828.856661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14833.794639] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14843.015296] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14848.689207] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14855.577956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14855.807716] Lustre: lustre-OST0000: new disk, initializing [14855.810978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14855.818808] Lustre: 475048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14855.909504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14861.560217] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14861.909402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14861.923811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14862.052359] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14871.204035] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14875.502994] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing lnet_if_list [14882.085106] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14882.152453] LNet: Added LNI 192.168.204.145@tcp10 [8/256/0/180] [14889.993070] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14890.051749] LNet: Added LNI 192.168.204.145@tcp11 [8/256/0/180] [14895.549102] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14895.589818] LNet: Added LNI 192.168.204.145@tcp12 [8/256/0/180] [14900.567872] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14900.593505] LNet: Added LNI 192.168.204.145@tcp13 [8/256/0/180] [14905.870554] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14905.916751] LNet: Added LNI 192.168.204.145@tcp14 [8/256/0/180] [14910.948195] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14910.979407] LNet: Added LNI 192.168.204.145@tcp15 [8/256/0/180] [14916.714796] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14916.763955] LNet: Added LNI 192.168.204.145@tcp16 [8/256/0/180] [14922.653909] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14922.707829] LNet: Added LNI 192.168.204.145@tcp17 [8/256/0/180] [14928.193362] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14928.219761] LNet: Added LNI 192.168.204.145@tcp18 [8/256/0/180] [14933.542871] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14933.583940] LNet: Added LNI 192.168.204.145@tcp19 [8/256/0/180] [14940.376475] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14940.405429] LNet: Added LNI 192.168.204.145@tcp20 [8/256/0/180] [14947.859815] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14947.884723] LNet: Added LNI 192.168.204.145@tcp21 [8/256/0/180] [14954.613541] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14954.660974] LNet: Added LNI 192.168.204.145@tcp22 [8/256/0/180] [14961.976186] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14962.019703] LNet: Added LNI 192.168.204.145@tcp23 [8/256/0/180] [14969.100266] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14969.130875] LNet: Added LNI 192.168.204.145@tcp24 [8/256/0/180] [14977.581486] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14977.652976] LNet: Added LNI 192.168.204.145@tcp25 [8/256/0/180] [14985.771398] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14985.817888] LNet: Added LNI 192.168.204.145@tcp26 [8/256/0/180] [14994.681917] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14994.733219] LNet: Added LNI 192.168.204.145@tcp27 [8/256/0/180] [15002.044640] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15002.073682] LNet: Added LNI 192.168.204.145@tcp28 [8/256/0/180] [15008.783866] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15008.808942] LNet: Added LNI 192.168.204.145@tcp29 [8/256/0/180] [15016.203697] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15016.250242] LNet: Added LNI 192.168.204.145@tcp30 [8/256/0/180] [15024.864172] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15024.922101] LNet: Added LNI 192.168.204.145@tcp31 [8/256/0/180] [15033.296460] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15033.356959] LNet: Added LNI 192.168.204.145@tcp32 [8/256/0/180] [15043.414355] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15043.452627] LNet: Added LNI 192.168.204.145@tcp33 [8/256/0/180] [15053.055879] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15053.121640] LNet: Added LNI 192.168.204.145@tcp34 [8/256/0/180] [15063.042549] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15063.100690] LNet: Added LNI 192.168.204.145@tcp35 [8/256/0/180] [15074.034218] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15074.097751] LNet: Added LNI 192.168.204.145@tcp36 [8/256/0/180] [15081.845942] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15081.906851] LNet: Added LNI 192.168.204.145@tcp37 [8/256/0/180] [15090.376087] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15090.397378] LNet: Added LNI 192.168.204.145@tcp38 [8/256/0/180] [15098.878861] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15098.930404] LNet: Added LNI 192.168.204.145@tcp39 [8/256/0/180] [15106.990126] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15107.040756] LNet: Added LNI 192.168.204.145@tcp40 [8/256/0/180] [15115.277600] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15115.308156] LNet: Added LNI 192.168.204.145@tcp41 [8/256/0/180] [15124.414493] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15124.442180] LNet: Added LNI 192.168.204.145@tcp42 [8/256/0/180] [15131.164474] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15131.190821] LNet: Added LNI 192.168.204.145@tcp43 [8/256/0/180] [15139.433798] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15139.468898] LNet: Added LNI 192.168.204.145@tcp44 [8/256/0/180] [15149.717555] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15149.741092] LNet: Added LNI 192.168.204.145@tcp45 [8/256/0/180] [15159.317731] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15159.360758] LNet: Added LNI 192.168.204.145@tcp46 [8/256/0/180] [15171.896734] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15171.933230] LNet: Added LNI 192.168.204.145@tcp47 [8/256/0/180] [15183.726157] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15183.764284] LNet: Added LNI 192.168.204.145@tcp48 [8/256/0/180] [15194.554524] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15194.583450] LNet: Added LNI 192.168.204.145@tcp49 [8/256/0/180] [15204.355936] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15204.394093] LNet: Added LNI 192.168.204.145@tcp50 [8/256/0/180] [15228.387163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15228.405862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15228.439940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15230.948393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15230.952307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15230.993970] Lustre: Skipped 2 previous similar messages [15234.368616] Lustre: server umount lustre-MDT0000 complete [15238.278591] LustreError: 473073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788608812 with bad export cookie 6855716281857968559 [15238.282142] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15238.295291] LustreError: 473073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15238.697814] Lustre: server umount lustre-MDT0001 complete [15257.570475] Lustre: 472351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788608815/real 1788608815] req@00000000041adc1e x1875491811420544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788608831 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15257.604448] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15257.615406] Lustre: Skipped 1 previous similar message [15257.834049] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [15259.685269] Lustre: 472351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788608817/real 1788608817] req@000000001551087e x1875491811420672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788608833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15262.688092] Lustre: 472353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788608820/real 1788608820] req@000000005715706a x1875491811420800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788608836 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15264.741905] Lustre: 472352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788608822/real 1788608822] req@0000000068ff2bcb x1875491811421056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788608838 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15268.716213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15269.856186] Lustre: 472351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788608827/real 1788608827] req@0000000068d4cf19 x1875491811421184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788608843 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15283.700035] LustreError: 475042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15283.738159] LustreError: 475042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15283.794740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15286.881796] LustreError: 475043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15286.896222] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15287.910181] LustreError: 475042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15288.979470] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15293.412390] LustreError: 475043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15298.538514] LustreError: 475042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15299.607253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15300.160607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15302.216785] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15305.481291] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15313.445122] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15320.367534] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15343.092952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15343.094899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15343.123984] Lustre: Skipped 1 previous similar message [15347.585416] Lustre: server umount lustre-OST0000 complete [15353.328559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15353.342980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15353.347040] Lustre: Skipped 2 previous similar messages [15358.396753] Lustre: server umount lustre-MDT0000 complete [15358.443263] LustreError: 499325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15358.463404] LustreError: 499325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15362.921962] LustreError: 473072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788608936 with bad export cookie 6855716281857971191 [15362.930614] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15362.932950] LustreError: 473072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15363.363407] Lustre: server umount lustre-MDT0001 complete [15374.311102] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [15377.079152] Key type lgssc unregistered [15377.665300] LNet: 502119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15377.674165] LNetError: 502119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15378.720439] LNet: Removed LNI 192.168.204.145@tcp [15378.725670] LNetError: 502119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15378.736561] LNet: Removed LNI 192.168.204.145@tcp10 [15378.744813] LNet: Removed LNI 192.168.204.145@tcp11 [15378.748992] LNet: Removed LNI 192.168.204.145@tcp12 [15378.759256] LNet: Removed LNI 192.168.204.145@tcp13 [15378.761146] LNet: Removed LNI 192.168.204.145@tcp14 [15378.768784] LNet: Removed LNI 192.168.204.145@tcp15 [15378.775977] LNet: Removed LNI 192.168.204.145@tcp16 [15378.781924] LNet: Removed LNI 192.168.204.145@tcp17 [15378.787984] LNet: Removed LNI 192.168.204.145@tcp18 [15378.794896] LNet: Removed LNI 192.168.204.145@tcp19 [15378.809618] LNet: Removed LNI 192.168.204.145@tcp20 [15378.812090] LNet: Removed LNI 192.168.204.145@tcp21 [15378.818764] LNet: Removed LNI 192.168.204.145@tcp22 [15378.821374] LNet: Removed LNI 192.168.204.145@tcp23 [15378.823227] LNet: Removed LNI 192.168.204.145@tcp24 [15378.828924] LNet: Removed LNI 192.168.204.145@tcp25 [15378.832875] LNet: Removed LNI 192.168.204.145@tcp26 [15378.835448] LNet: Removed LNI 192.168.204.145@tcp27 [15378.838403] LNet: Removed LNI 192.168.204.145@tcp28 [15378.841720] LNet: Removed LNI 192.168.204.145@tcp29 [15378.844888] LNet: Removed LNI 192.168.204.145@tcp30 [15378.847729] LNet: Removed LNI 192.168.204.145@tcp31 [15378.849988] LNet: Removed LNI 192.168.204.145@tcp32 [15378.852398] LNet: Removed LNI 192.168.204.145@tcp33 [15378.855272] LNet: Removed LNI 192.168.204.145@tcp34 [15378.857841] LNet: Removed LNI 192.168.204.145@tcp35 [15378.860543] LNet: Removed LNI 192.168.204.145@tcp36 [15378.862434] LNet: Removed LNI 192.168.204.145@tcp37 [15378.864451] LNet: Removed LNI 192.168.204.145@tcp38 [15378.868577] LNet: Removed LNI 192.168.204.145@tcp39 [15378.870827] LNet: Removed LNI 192.168.204.145@tcp40 [15378.873410] LNet: Removed LNI 192.168.204.145@tcp41 [15378.875815] LNet: Removed LNI 192.168.204.145@tcp42 [15378.878397] LNet: Removed LNI 192.168.204.145@tcp43 [15378.880764] LNet: Removed LNI 192.168.204.145@tcp44 [15378.883572] LNet: Removed LNI 192.168.204.145@tcp45 [15378.886141] LNet: Removed LNI 192.168.204.145@tcp46 [15378.888368] LNet: Removed LNI 192.168.204.145@tcp47 [15378.891603] LNet: Removed LNI 192.168.204.145@tcp48 [15378.894231] LNet: Removed LNI 192.168.204.145@tcp49 [15378.906394] LNet: Removed LNI 192.168.204.145@tcp50 [15379.928843] Key type .llcrypt unregistered [15379.931576] Key type ._llcrypt unregistered [15403.780853] Key type ._llcrypt registered [15403.783841] Key type .llcrypt registered [15403.881353] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [15420.574807] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [15421.779162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15421.853036] alg: No test for adler32 (adler32-zlib) [15423.203937] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [15423.470937] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [15425.304661] Key type lgssc registered [15426.685746] Lustre: Echo OBD driver; http://www.lustre.org/ [15466.631992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15466.676797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15467.843295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15467.872677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15467.977407] Lustre: lustre-MDT0000: new disk, initializing [15468.079636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15468.107564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15472.143968] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15483.194613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15483.362754] Lustre: 505318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15483.421120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15483.424777] Lustre: Skipped 1 previous similar message [15483.519251] Lustre: lustre-MDT0001: new disk, initializing [15483.612641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15483.647111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15483.651766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15488.382303] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15499.995244] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15506.777293] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15516.756537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15516.969646] Lustre: *** cfs_fail_loc=90f, val=0*** [15517.092881] Lustre: lustre-OST0000: new disk, initializing [15517.103035] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15517.120519] Lustre: 507335:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15517.259241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15524.531975] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15526.941117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15526.977139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15527.108076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15533.612292] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15542.244657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15542.259717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15542.267425] Lustre: Skipped 1 previous similar message [15542.290194] Lustre: Skipped 1 previous similar message [15547.368738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15547.381569] Lustre: Skipped 1 previous similar message [15548.090806] Lustre: server umount lustre-OST0000 complete [15552.483569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15552.487782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15552.502776] Lustre: Skipped 1 previous similar message [15552.522822] Lustre: Skipped 1 previous similar message [15557.603345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15557.615341] Lustre: Skipped 1 previous similar message [15558.521990] Lustre: server umount lustre-MDT0000 complete [15562.055161] LustreError: 507334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788609135 with bad export cookie 5096796603843323418 [15562.057846] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15562.062492] LustreError: 507334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15562.409960] Lustre: server umount lustre-MDT0001 complete [15574.225608] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [15577.895489] Key type lgssc unregistered [15578.251037] LNet: 509240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15578.269268] LNetError: 509240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15579.304392] LNet: Removed LNI 192.168.204.145@tcp [15580.377387] Key type .llcrypt unregistered [15580.380456] Key type ._llcrypt unregistered [15601.921272] Key type ._llcrypt registered [15601.924780] Key type .llcrypt registered [15602.078374] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [15603.631865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15603.641912] alg: No test for adler32 (adler32-zlib) [15604.809444] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [15605.093118] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [15606.880377] Key type lgssc registered [15608.470411] Lustre: Echo OBD driver; http://www.lustre.org/ [15616.732111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15616.751358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15618.362901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15623.130105] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15628.782935] LustreError: 510509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15628.812608] LustreError: 510509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15632.695596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15633.252556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15638.763807] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15646.700629] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15653.389040] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15660.951215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15661.423542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15668.685611] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15675.473310] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15687.475244] loop0: detected capacity change from 0 to 204800000 [15687.699270] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15687.716255] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15687.747191] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15694.373789] loop0: detected capacity change from 0 to 204800000 [15694.591240] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15694.599901] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15694.608526] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15699.437707] loop0: detected capacity change from 0 to 204800000 [15704.422555] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15704.516556] Lustre: fs15246-OST0000: new disk, initializing [15704.520571] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15704.527737] Lustre: 514108:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15704.606945] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15710.517614] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15720.462558] loop1: detected capacity change from 0 to 204800000 [15724.727597] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15724.797111] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15724.853960] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15724.906660] Lustre: fs15246-MDT0000: new disk, initializing [15725.073812] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15725.089404] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15725.167279] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15726.241336] Lustre: 515229:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000078d47923 x1875492657477760/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788609316 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15729.247162] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15735.347195] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15735.381038] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15735.417174] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15757.798124] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15757.817400] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15762.044074] Lustre: server umount fs15246-MDT0000 complete [15762.914808] LustreError: 513476:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15768.039554] LustreError: 510503:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15770.955485] Lustre: server umount fs15246-OST0000 complete [15785.442996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15785.451728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15785.466362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15786.465297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15786.470997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15786.490696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15788.536914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15788.548664] Lustre: Skipped 1 previous similar message [15790.292647] Lustre: server umount lustre-OST0000 complete [15797.218843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15797.221938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15797.247613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15800.099302] Lustre: server umount lustre-MDT0000 complete [15803.832988] LustreError: 510489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788609377 with bad export cookie 11341485026623411044 [15803.837015] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15803.848016] LustreError: 510489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15804.271412] Lustre: server umount lustre-MDT0001 complete [15816.241418] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [15819.156317] Key type lgssc unregistered [15819.519763] LNet: 518328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15819.533056] LNetError: 518328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15819.563383] LNet: Removed LNI 192.168.204.145@tcp [15820.641345] Key type .llcrypt unregistered [15820.643901] Key type ._llcrypt unregistered [15841.735351] Key type ._llcrypt registered [15841.738102] Key type .llcrypt registered [15841.861563] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [15843.923939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15844.031078] alg: No test for adler32 (adler32-zlib) [15845.040432] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [15845.271284] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [15846.984140] Key type lgssc registered [15848.725197] Lustre: Echo OBD driver; http://www.lustre.org/ [15887.976813] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [15897.320220] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [15928.174939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15928.205471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15929.588236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15929.667673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15929.749723] Lustre: lustre-MDT0000: new disk, initializing [15929.826840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15929.843076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15934.258688] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15944.189321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15944.283144] Lustre: 522792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15944.333469] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15944.336852] Lustre: Skipped 1 previous similar message [15944.460265] Lustre: lustre-MDT0001: new disk, initializing [15944.564892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15944.598270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15944.619094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15949.025748] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15957.564473] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15963.117744] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15970.630807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15970.972810] Lustre: lustre-OST0000: new disk, initializing [15970.980527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15970.988145] Lustre: 524758:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15971.084395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15977.781522] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15980.623873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15980.639299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15980.774455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15987.844359] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15993.467969] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15993.661245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15999.180805] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15999.398219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16001.507345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16001.511931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16001.513092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16001.537428] Lustre: Skipped 1 previous similar message [16001.554548] LustreError: Skipped 1 previous similar message [16006.626828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16006.636383] Lustre: Skipped 2 previous similar messages [16007.510107] Lustre: server umount lustre-OST0000 complete [16011.233574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16011.246595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16011.273860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16016.866829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16016.876861] Lustre: Skipped 2 previous similar messages [16017.395803] Lustre: server umount lustre-MDT0000 complete [16020.816631] LustreError: 526097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788609594 with bad export cookie 1131686658106443185 [16020.821553] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16020.828908] LustreError: 526097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16021.173075] Lustre: server umount lustre-MDT0001 complete [16043.858824] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [16054.158817] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [16086.161713] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15791 sec ========= 08:00:58 (1788609658) [16088.037381] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:00:59 (1788609659) === [16091.551257] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:01:03 (1788609663) === [16133.064502] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [16135.863040] Key type lgssc unregistered [16136.088913] LNet: 531412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16136.095515] LNetError: 531412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16136.111431] LNet: Removed LNI 192.168.204.145@tcp [16136.796727] Key type .llcrypt unregistered [16136.800220] Key type ._llcrypt unregistered