[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 461017692 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003165] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008401] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011147] LSM: Security Framework initializing [ 0.012062] Yama: becoming mindful. [ 0.013046] SELinux: Initializing. [ 0.014068] *** VALIDATE selinux *** [ 0.023436] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028112] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029168] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030147] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031130] *** VALIDATE tmpfs *** [ 0.032488] *** VALIDATE proc *** [ 0.034171] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.037178] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040033] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.044637] debug: unmapping init [mem 0xffffffffa2059000-0xffffffffa2060fff] [ 0.046185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047757] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048026] ... version: 2 [ 0.049014] ... bit width: 48 [ 0.050013] ... generic registers: 4 [ 0.051017] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053018] ... fixed-purpose events: 3 [ 0.054015] ... event mask: 000000070000000f [ 0.055347] rcu: Hierarchical SRCU implementation. [ 0.057518] smp: Bringing up secondary CPUs ... [ 0.058569] x86: Booting SMP configuration: [ 0.059026] .... node #0, CPUs: #1 #2 #3 [ 0.063011] smp: Brought up 1 node, 4 CPUs [ 0.065017] smpboot: Max logical packages: 1 [ 0.066013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149432] node 0 deferred pages initialised in 80ms [ 0.151428] devtmpfs: initialized [ 0.152265] x86/mm: Memory block size: 128MB [ 0.154720] gcov: version magic: 0x41383552 [ 0.156325] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158278] pinctrl core: initialized pinctrl subsystem [ 0.159214] [ 0.159814] ************************************************************* [ 0.160015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161018] ** ** [ 0.162013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163016] ** ** [ 0.164014] ** This means that this kernel is built to expose internal ** [ 0.165019] ** IOMMU data structures, which may compromise security on ** [ 0.166019] ** your system. ** [ 0.167019] ** ** [ 0.168021] ** If you see this message and you are not debugging the ** [ 0.169014] ** kernel, report this immediately to your vendor! ** [ 0.170020] ** ** [ 0.171022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172018] ************************************************************* [ 0.173972] NET: Registered protocol family 16 [ 0.174394] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175107] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177350] cpuidle: using governor menu [ 0.179835] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182566] PCI: Using configuration type 1 for base access [ 0.185139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194039] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196073] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198290] cryptd: max_cpu_qlen set to 1000 [ 0.201255] ACPI: Added _OSI(Module Device) [ 0.203016] ACPI: Added _OSI(Processor Device) [ 0.205015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.212000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.217615] ACPI: Interpreter enabled [ 0.218057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.220011] ACPI: Using IOAPIC for interrupt routing [ 0.222106] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.225589] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.236000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.238045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.241022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244146] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.249322] acpiphp: Slot [2] registered [ 0.251093] acpiphp: Slot [5] registered [ 0.253116] acpiphp: Slot [6] registered [ 0.254127] acpiphp: Slot [7] registered [ 0.256126] acpiphp: Slot [8] registered [ 0.258213] acpiphp: Slot [9] registered [ 0.259331] acpiphp: Slot [10] registered [ 0.262149] acpiphp: Slot [3] registered [ 0.263103] acpiphp: Slot [4] registered [ 0.265137] acpiphp: Slot [11] registered [ 0.267097] acpiphp: Slot [12] registered [ 0.268096] acpiphp: Slot [13] registered [ 0.270114] acpiphp: Slot [14] registered [ 0.272111] acpiphp: Slot [15] registered [ 0.273109] acpiphp: Slot [16] registered [ 0.276105] acpiphp: Slot [17] registered [ 0.278129] acpiphp: Slot [18] registered [ 0.279098] acpiphp: Slot [19] registered [ 0.281126] acpiphp: Slot [20] registered [ 0.282104] acpiphp: Slot [21] registered [ 0.284115] acpiphp: Slot [22] registered [ 0.286152] acpiphp: Slot [23] registered [ 0.287099] acpiphp: Slot [24] registered [ 0.289082] acpiphp: Slot [25] registered [ 0.290155] acpiphp: Slot [26] registered [ 0.291187] acpiphp: Slot [27] registered [ 0.293116] acpiphp: Slot [28] registered [ 0.294119] acpiphp: Slot [29] registered [ 0.296172] acpiphp: Slot [30] registered [ 0.297096] acpiphp: Slot [31] registered [ 0.299104] PCI host bridge to bus 0000:00 [ 0.301030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.303026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.306039] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.308024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.310030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.313033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.315193] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.317996] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.322438] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.331766] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.336562] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.339020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.342023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.345023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.348566] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.350857] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.354047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.356972] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.363019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.376016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.382027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.388379] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.396020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.409022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.433020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.449107] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.489020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.499555] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.506020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.513018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.528027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.539727] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.545015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.549016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.562020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.573454] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.579021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.589015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.607019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.615555] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.621015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.627017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.638024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.648063] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.649405] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.651376] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.653204] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.655165] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.659056] iommu: Default domain type: Passthrough [ 0.660391] SCSI subsystem initialized [ 0.661153] ACPI: bus type USB registered [ 0.663110] usbcore: registered new interface driver usbfs [ 0.665083] usbcore: registered new interface driver hub [ 0.667103] usbcore: registered new device driver usb [ 0.670204] pps_core: LinuxPPS API ver. 1 registered [ 0.672012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.675065] PTP clock support registered [ 0.677152] EDAC MC: Ver: 3.0.0 [ 0.679466] PCI: Using ACPI for IRQ routing [ 0.681595] NetLabel: Initializing [ 0.682007] NetLabel: domain hash size = 128 [ 0.683008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.684063] NetLabel: unlabeled traffic allowed by default [ 0.685149] vgaarb: loaded [ 0.686317] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.688018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.693316] clocksource: Switched to clocksource kvm-clock [ 0.778343] VFS: Disk quotas dquot_6.6.0 [ 0.780084] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.782110] *** VALIDATE ramfs *** [ 0.783362] *** VALIDATE hugetlbfs *** [ 0.784777] pnp: PnP ACPI init [ 0.787149] pnp: PnP ACPI: found 6 devices [ 0.814624] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.817842] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.819758] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.821970] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.824370] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.826050] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.828735] NET: Registered protocol family 2 [ 0.831230] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.834711] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.836836] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.840977] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.844122] TCP: Hash tables configured (established 65536 bind 65536) [ 0.846509] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.848751] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.850872] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.854199] NET: Registered protocol family 1 [ 0.857125] RPC: Registered named UNIX socket transport module. [ 0.859645] RPC: Registered udp transport module. [ 0.862069] RPC: Registered tcp transport module. [ 0.863755] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.865881] NET: Registered protocol family 44 [ 0.867556] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.869594] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.871622] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.873847] PCI: CLS 0 bytes, default 64 [ 0.875458] Unpacking initramfs... [ 2.278572] debug: unmapping init [mem 0xffff9d2c3cc54000-0xffff9d2c3ffbffff] [ 2.282543] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.284944] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.287780] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.726467] Initialise system trusted keyrings [ 2.728123] Key type blacklist registered [ 2.730110] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.740301] zbud: loaded [ 2.743974] *** VALIDATE nfs *** [ 2.745423] *** VALIDATE nfs4 *** [ 2.746869] pstore: using deflate compression [ 2.751505] Platform Keyring initialized [ 2.822215] NET: Registered protocol family 38 [ 2.823564] Key type asymmetric registered [ 2.824462] Asymmetric key parser 'x509' registered [ 2.825450] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.827188] io scheduler mq-deadline registered [ 2.828205] io scheduler kyber registered [ 2.829138] io scheduler bfq registered [ 2.830118] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.831599] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.833341] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.834941] ACPI: Power Button [PWRF] [ 2.838712] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.843129] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.850661] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.856117] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.867188] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.893410] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.919400] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.924294] Non-volatile memory driver v1.3 [ 2.925740] Linux agpgart interface v0.103 [ 2.956661] virtio_blk virtio1: [vda] 150064 512-byte logical blocks (76.8 MB/73.3 MiB) [ 2.959744] vda: detected capacity change from 0 to 76832768 [ 2.983414] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.985296] vdb: detected capacity change from 0 to 1073741824 [ 3.006285] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.008891] vdc: detected capacity change from 0 to 2621440000 [ 3.027465] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.029464] vdd: detected capacity change from 0 to 2621440000 [ 3.041602] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.044766] vde: detected capacity change from 0 to 4294967296 [ 3.064192] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.067403] vdf: detected capacity change from 0 to 4294967296 [ 3.073788] libphy: Fixed MDIO Bus: probed [ 3.080514] usbcore: registered new interface driver usbserial_generic [ 3.082415] usbserial: USB Serial support registered for generic [ 3.083893] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.086902] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.088113] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.090481] mousedev: PS/2 mouse device common for all mice [ 3.095440] rtc_cmos 00:05: RTC can wake from S4 [ 3.097448] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.099444] rtc_cmos 00:05: registered as rtc0 [ 3.099517] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.099553] intel_pstate: CPU model not supported [ 3.104242] hid: raw HID events driver (C) Jiri Kosina [ 3.114660] usbcore: registered new interface driver usbhid [ 3.114946] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.116736] usbhid: USB HID core driver [ 3.116898] drop_monitor: Initializing network drop monitor service [ 3.123251] Initializing XFRM netlink socket [ 3.125520] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.125535] NET: Registered protocol family 10 [ 3.130991] Segment Routing with IPv6 [ 3.132330] NET: Registered protocol family 17 [ 3.135057] mpls_gso: MPLS GSO support [ 3.141973] RAS: Correctable Errors collector initialized. [ 3.143709] AVX version of gcm_enc/dec engaged. [ 3.144639] AES CTR mode by8 optimization enabled [ 3.210607] sched_clock: Marking stable (3210583588, 0)->(4186112398, -975528810) [ 3.214982] registered taskstats version 1 [ 3.216216] Loading compiled-in X.509 certificates [ 3.218274] zswap: loaded using pool lzo/zbud [ 3.242632] Key type big_key registered [ 3.254706] Key type encrypted registered [ 3.256058] ima: No TPM chip found, activating TPM-bypass! [ 3.258747] ima: Allocated hash algorithm: sha1 [ 3.260663] ima: No architecture policies found [ 3.262761] evm: Initialising EVM extended attributes: [ 3.264556] evm: security.selinux [ 3.265541] evm: security.ima [ 3.267042] evm: security.capability [ 3.268451] evm: HMAC attrs: 0x1 [ 3.270841] rtc_cmos 00:05: setting system clock to 2026-09-07 02:35:15 UTC (1788748515) [ 3.276090] debug: unmapping init [mem 0xffffffffa3003000-0xffffffffa31fffff] [ 3.278805] debug: unmapping init [mem 0xffffffffa1d82000-0xffffffffa2058fff] [ 3.287082] Write protecting the kernel read-only data: 28672k [ 3.290436] debug: unmapping init [mem 0xffffffffa0403000-0xffffffffa05fffff] [ 3.293979] debug: unmapping init [mem 0xffffffffa0d14000-0xffffffffa0dfffff] [ 3.327982] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.334265] systemd[1]: Detected virtualization kvm. [ 3.335667] systemd[1]: Detected architecture x86-64. [ 3.336940] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.358731] systemd[1]: No hostname configured. [ 3.360567] systemd[1]: Set hostname to . [ 3.362651] random: systemd: uninitialized urandom read (16 bytes read) [ 3.365178] systemd[1]: Initializing machine ID from random generator. [ 3.410829] random: ln: uninitialized urandom read (6 bytes read) [ 3.467487] random: systemd: uninitialized urandom read (16 bytes read) [ 3.470633] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.475901] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.479051] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.941850] device-mapper: uevent: version 1.0.3 [ 3.943798] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.548159] random: fast init done [ 4.551066] virtio_net virtio0 ens2: renamed from eth0 [ 4.627525] scsi host0: ata_piix [ 4.657552] scsi host1: ata_piix [ 4.659049] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.661511] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.774186] random: crng init done [ 8.775600] random: 7 urandom warning(s) missed due to ratelimiting [ 8.821146] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.355861] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.473083] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.737700] SELinux: Disabled at runtime. [ 10.797212] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.803228] systemd[1]: Detected virtualization kvm. [ 10.804477] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.254584] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.257696] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.263187] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.267762] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.271748] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.281942] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.287803] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ 11.408708] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 11.761439] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.063905] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.110548] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.261100] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.275830] EDAC sbridge: Ver: 1.1.2 [ 13.921860] Key type dns_resolver registered [ 14.219743] NFS: Registering the id_resolver key type [ 14.221694] Key type id_resolver registered [ 14.223102] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg226-server login: [ 45.237775] libcfs: loading out-of-tree module taints kernel. [ 45.367436] Key type ._llcrypt registered [ 45.372273] Key type .llcrypt registered [ 45.498511] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 68.942981] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 70.456987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.505663] alg: No test for adler32 (adler32-zlib) [ 72.222847] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 73.284353] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 75.289323] Key type lgssc registered [ 77.558411] hrtimer: interrupt took 3389430 ns [ 77.579001] Lustre: Echo OBD driver; http://www.lustre.org/ [ 97.425826] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 140.269858] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 153.230989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 153.281180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 154.667620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 154.712753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 154.817939] Lustre: lustre-MDT0000: new disk, initializing [ 154.917396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 154.943860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 160.023602] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.092382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.199749] Lustre: 6505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 173.231449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 173.236292] Lustre: Skipped 1 previous similar message [ 173.316394] Lustre: lustre-MDT0001: new disk, initializing [ 173.379741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 173.412310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 173.432826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 178.510858] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.647445] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 193.792900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.091566] Lustre: lustre-OST0000: new disk, initializing [ 194.094843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 194.100352] Lustre: 8443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.163925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.672670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 198.688356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 198.767088] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.892713] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.267278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.393857] Lustre: lustre-OST0001: new disk, initializing [ 212.400661] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 212.406298] Lustre: 9514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.457399] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 218.137749] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.716134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 221.722530] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 221.787831] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 228.554134] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 233.668935] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 239.314239] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing check_logdir /tmp/testlogs/ [ 244.328634] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing yml_node [ 247.426300] Lustre: DEBUG MARKER: Client: 2.17.58.41 [ 249.457372] Lustre: DEBUG MARKER: MDS: 2.17.58.41 [ 252.013701] Lustre: DEBUG MARKER: OSS: 2.17.58.41 [ 253.671828] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 6 22:39:24 EDT 2026 [ 269.056524] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 270.316815] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 286.179840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 286.186061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.200200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.224254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.225386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.239693] Lustre: Skipped 1 previous similar message [ 288.245669] Lustre: Skipped 1 previous similar message [ 292.265500] Lustre: server umount lustre-MDT0000 complete [ 295.281335] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748807 with bad export cookie 4873768254229129747 [ 295.293400] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 298.466436] LustreError: 10187:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.480473] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.492405] LustreError: 10187:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 298.521164] Lustre: Skipped 1 previous similar message [ 298.529457] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.537271] Lustre: Skipped 1 previous similar message [ 300.512166] LustreError: 10187:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.530457] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.533270] Lustre: Skipped 1 previous similar message [ 301.785403] Lustre: server umount lustre-MDT0001 complete [ 305.567367] Lustre: server umount lustre-OST0000 complete [ 309.027321] Lustre: server umount lustre-OST0001 complete [ 315.196981] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 324.120955] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 353.020511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 353.230736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 353.254405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 353.321619] Lustre: lustre-MDT0000: new disk, initializing [ 353.380351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 353.391282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 357.028668] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 366.977817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.215637] Lustre: 15065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 367.315344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 367.320151] Lustre: Skipped 1 previous similar message [ 367.438941] Lustre: lustre-MDT0001: new disk, initializing [ 367.627064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 367.707332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 367.716821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 371.893801] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 380.635532] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 385.984392] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 392.070865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 392.409986] Lustre: lustre-OST0000: new disk, initializing [ 392.413410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 392.420987] Lustre: 17034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 392.512577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 393.817225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 393.829901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 394.001730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 398.483775] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 406.153792] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 410.102067] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 410.267583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 414.621769] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 414.835826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 419.810878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 419.811163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 419.816350] LustreError: Skipped 1 previous similar message [ 419.817382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 419.834773] Lustre: Skipped 2 previous similar messages [ 419.859923] Lustre: Skipped 1 previous similar message [ 422.637612] Lustre: server umount lustre-OST0000 complete [ 429.032469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 429.049462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 429.074438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.287238] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 440.649849] Lustre: server umount lustre-MDT0000 complete [ 444.305321] LustreError: 15860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748956 with bad export cookie 4873768254229131280 [ 444.318064] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 444.328171] LustreError: 15860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 444.558840] Lustre: server umount lustre-MDT0001 complete [ 448.528928] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:42:39 (1788748959) [ 450.849335] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 452.722987] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:42:43 (1788748963) [ 459.177206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.848822] LustreError: 19444:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 459.875415] LustreError: 19444:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 459.956145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.494477] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 465.378024] LustreError: 19445:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 470.497824] LustreError: 19444:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.783515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 477.582559] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 484.636658] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 489.995762] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 496.389415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 502.386289] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 509.246460] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 515.040964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 515.054105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.058798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 515.066359] Lustre: Skipped 2 previous similar messages [ 515.071333] Lustre: Skipped 6 previous similar messages [ 517.047771] Lustre: server umount lustre-OST0000 complete [ 526.437223] Lustre: server umount lustre-MDT0000 complete [ 529.471121] LustreError: 19426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749041 with bad export cookie 4873768254229132162 [ 529.484045] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.485539] LustreError: 19426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 529.727121] Lustre: server umount lustre-MDT0001 complete [ 539.123510] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 542.116629] Key type lgssc unregistered [ 542.386710] LNet: 22972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.397571] LNetError: 22972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 542.414407] LNet: Removed LNI 192.168.202.126@tcp [ 543.332373] Key type .llcrypt unregistered [ 543.338326] Key type ._llcrypt unregistered [ 564.435523] Key type ._llcrypt registered [ 564.437806] Key type .llcrypt registered [ 564.548950] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 565.219788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 565.258354] alg: No test for adler32 (adler32-zlib) [ 566.399348] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 566.598685] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 568.279187] Key type lgssc registered [ 569.213696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 576.519709] loop: module loaded [ 576.625185] loop0: detected capacity change from 0 to 204800000 [ 576.854204] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 576.875624] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 577.248853] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 583.008856] loop0: detected capacity change from 0 to 204800000 [ 583.216984] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 583.238179] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 583.265114] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 588.238497] loop0: detected capacity change from 0 to 204800000 [ 593.492098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 593.546230] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.842459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 594.883714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 594.987503] Lustre: lustre-MDT0000: new disk, initializing [ 595.105863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 595.128582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 599.770947] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 608.459551] loop1: detected capacity change from 0 to 204800000 [ 613.634784] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 613.979371] Lustre: lustre-OST07c6: new disk, initializing [ 613.982917] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 613.988716] Lustre: Skipped 1 previous similar message [ 613.996407] Lustre: 26153:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 614.069760] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 620.133550] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 621.701431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 621.711980] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 621.752067] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 625.766937] Lustre: 25141:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 625.773251] Lustre: 25141:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 625.778210] Lustre: 25141:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 625.784040] Lustre: 25141:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 625.789479] Lustre: 25141:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 625.795841] Lustre: 25141:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 643.268102] Lustre: 25140:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 643.278935] Lustre: 25140:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 643.283790] Lustre: 25140:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 643.291305] Lustre: 25140:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 643.300836] Lustre: 25140:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 643.308626] Lustre: 25140:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 647.731289] Lustre: 25142:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 647.736597] Lustre: 25142:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 647.740368] Lustre: 25142:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 647.746120] Lustre: 25142:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 647.750915] Lustre: 25142:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 647.759395] Lustre: 25142:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 647.763163] Lustre: 25142:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 647.766401] Lustre: 25142:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 647.769152] Lustre: 25142:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 647.774604] Lustre: 25142:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 647.780524] Lustre: 25142:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 647.793525] Lustre: 25142:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 651.013937] Lustre: 25141:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 651.035747] Lustre: 25141:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 651.046902] Lustre: 25141:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 651.053936] Lustre: 25141:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 651.059788] Lustre: 25141:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 651.063328] Lustre: 25141:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 651.075341] Lustre: 25141:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 651.088593] Lustre: 25141:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 651.099252] Lustre: 25141:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 651.116604] Lustre: 25141:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 651.127651] Lustre: 25141:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 651.134324] Lustre: 25141:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 655.702154] Lustre: 25140:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 655.720837] Lustre: 25140:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 655.727577] Lustre: 25140:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 655.739912] Lustre: 25140:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 655.748685] Lustre: 25140:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 655.758824] Lustre: 25140:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 655.772755] Lustre: 25140:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 655.782544] Lustre: 25140:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 655.787910] Lustre: 25140:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 655.793752] Lustre: 25140:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 655.802062] Lustre: 25140:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 655.811717] Lustre: 25140:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 666.389142] Lustre: 26161:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 666.395224] Lustre: 26161:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 666.397726] Lustre: 26161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 666.401591] Lustre: 26161:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 666.405451] Lustre: 26161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 666.413413] Lustre: 26161:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 666.417386] Lustre: 26161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 666.423160] Lustre: 26161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 666.427661] Lustre: 26161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 666.430869] Lustre: 26161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 666.442590] Lustre: 26161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 666.452146] Lustre: 26161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 713.697202] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 713.708195] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.727447] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 718.817030] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 719.315832] Lustre: server umount lustre-OST07c6 complete [ 727.659267] Lustre: server umount lustre-MDT0000 complete [ 742.354412] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 745.045991] Key type lgssc unregistered [ 745.315412] LNet: 28711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.323535] LNetError: 28711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.339449] LNet: Removed LNI 192.168.202.126@tcp [ 746.053579] Key type .llcrypt unregistered [ 746.057252] Key type ._llcrypt unregistered [ 762.344036] Key type ._llcrypt registered [ 762.345102] Key type .llcrypt registered [ 762.408462] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 763.415970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 763.430120] alg: No test for adler32 (adler32-zlib) [ 764.436816] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 764.676865] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 766.327835] Key type lgssc registered [ 767.190229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 773.503747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 773.522336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 774.940723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 778.877122] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 785.384638] LustreError: 29972:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 785.390880] LustreError: 29972:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 785.594085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 785.949112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 790.346584] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 796.992745] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 801.361749] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 807.942348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 808.248054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 813.118711] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 818.140509] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 820.530556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 836.068659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 836.069384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 836.097246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 841.195580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 841.201420] Lustre: Skipped 1 previous similar message [ 841.376184] Lustre: server umount lustre-OST0000 complete [ 846.310821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 846.320922] Lustre: Skipped 1 previous similar message [ 846.325526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 846.329619] Lustre: Skipped 2 previous similar messages [ 850.413801] Lustre: server umount lustre-MDT0000 complete [ 851.425704] LustreError: 29969:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 851.436167] LustreError: 29969:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 852.993401] LustreError: 29952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749365 with bad export cookie 17598532078260999274 [ 852.994463] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 853.001662] LustreError: 29952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 853.358396] Lustre: server umount lustre-MDT0001 complete [ 862.529222] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 865.048772] Key type lgssc unregistered [ 865.322594] LNet: 33705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.325836] LNetError: 33705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.338257] LNet: Removed LNI 192.168.202.126@tcp [ 866.075431] Key type .llcrypt unregistered [ 866.077677] Key type ._llcrypt unregistered [ 880.735127] Key type ._llcrypt registered [ 880.739907] Key type .llcrypt registered [ 880.856311] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 881.502176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 881.522090] alg: No test for adler32 (adler32-zlib) [ 882.536741] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 882.747326] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 884.383240] Key type lgssc registered [ 885.070651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 891.524233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 891.538890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.910884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 896.828140] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 903.137905] LustreError: 34949:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 903.161726] LustreError: 34949:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 904.121880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.815099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 908.633359] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 915.115651] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 919.814494] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 925.917865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 926.196916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.049441] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 938.495576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 938.798418] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 948.728207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 948.732844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.738550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 948.763506] Lustre: Skipped 1 previous similar message [ 953.828362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.838079] Lustre: Skipped 2 previous similar messages [ 954.146887] Lustre: server umount lustre-OST0000 complete [ 958.946125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.949712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 958.957364] Lustre: Skipped 1 previous similar message [ 958.966995] Lustre: Skipped 1 previous similar message [ 963.597797] Lustre: server umount lustre-MDT0000 complete [ 964.071171] LustreError: 36192:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 964.096736] LustreError: 36192:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 966.600265] LustreError: 34932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749478 with bad export cookie 11802584886465083392 [ 966.609518] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 966.937982] Lustre: server umount lustre-MDT0001 complete [ 975.269454] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 977.283289] Key type lgssc unregistered [ 977.513955] LNet: 38483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.519264] LNetError: 38483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.529620] LNet: Removed LNI 192.168.202.126@tcp [ 978.264194] Key type .llcrypt unregistered [ 978.265618] Key type ._llcrypt unregistered [ 996.993384] Key type ._llcrypt registered [ 996.995588] Key type .llcrypt registered [ 997.082412] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 997.770634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 997.818902] alg: No test for adler32 (adler32-zlib) [ 998.990172] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 999.212710] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 1000.895277] Key type lgssc registered [ 1002.152650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1008.556467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1008.570475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.924901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.828992] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1018.654977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1019.066433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1023.182809] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1033.427568] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1043.898885] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1056.443694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1057.653606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.150315] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1076.443761] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1080.185548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1088.244973] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.26@tcp (stopping) [ 1089.681701] Lustre: server umount lustre-MDT0000 complete [ 1090.537456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1090.549676] LustreError: 39872:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1090.555280] Lustre: Skipped 1 previous similar message [ 1090.588961] LustreError: 39872:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 1093.179143] LustreError: 39857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749605 with bad export cookie 1975706105270432817 [ 1093.188732] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1093.325079] LustreError: 39871:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1093.379517] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.26@tcp (stopping) [ 1095.651647] LustreError: 39872:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1095.653616] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1095.669381] Lustre: Skipped 1 previous similar message [ 1095.672704] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1098.445360] LustreError: 40315:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.479773] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.26@tcp (stopping) [ 1099.516147] Lustre: server umount lustre-MDT0001 complete [ 1121.008971] Lustre: server umount lustre-OST0000 complete [ 1137.102254] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 1139.859977] Key type lgssc unregistered [ 1140.152773] LNet: 43739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.156864] LNetError: 43739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1140.171103] LNet: Removed LNI 192.168.202.126@tcp [ 1141.063981] Key type .llcrypt unregistered [ 1141.065670] Key type ._llcrypt unregistered [ 1164.392765] Key type ._llcrypt registered [ 1164.397156] Key type .llcrypt registered [ 1164.504786] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 1165.476339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1165.489596] alg: No test for adler32 (adler32-zlib) [ 1166.595878] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 1166.917055] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 1168.623432] Key type lgssc registered [ 1169.633622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1176.329077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1176.349054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.747352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1182.339200] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1187.300222] LustreError: 45151:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1187.314819] LustreError: 45151:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1190.026039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.505756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1195.123942] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1202.768430] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1207.820289] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1215.285552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1215.595477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1222.102375] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1222.772407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1229.780577] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1240.556531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1240.560759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.580438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1240.592299] Lustre: Skipped 1 previous similar message [ 1240.615412] Lustre: Skipped 1 previous similar message [ 1241.842080] Lustre: lustre-OST0000: Not available for connect from 192.168.202.26@tcp (stopping) [ 1244.822738] Lustre: server umount lustre-OST0000 complete [ 1265.123574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1265.138194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1265.142134] Lustre: Skipped 1 previous similar message [ 1269.548802] Lustre: server umount lustre-MDT0000 complete [ 1270.241273] LustreError: 45151:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1270.255589] LustreError: 45151:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1272.977197] LustreError: 45132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749785 with bad export cookie 12487665157073141256 [ 1272.993022] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1272.998611] LustreError: 45132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1279.469861] Lustre: server umount lustre-MDT0001 complete [ 1288.606145] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 1291.604977] Key type lgssc unregistered [ 1291.886473] LNet: 48859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1291.891575] LNetError: 48859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1291.902562] LNet: Removed LNI 192.168.202.126@tcp [ 1292.751232] Key type .llcrypt unregistered [ 1292.753381] Key type ._llcrypt unregistered [ 1314.325550] Key type ._llcrypt registered [ 1314.327919] Key type .llcrypt registered [ 1314.417393] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 1315.601946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1315.629589] alg: No test for adler32 (adler32-zlib) [ 1316.637685] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 1316.937340] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 1318.639188] Key type lgssc registered [ 1319.636623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1327.477539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1327.501646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.920338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.290254] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1339.362706] LustreError: 50274:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.385963] LustreError: 50274:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1340.443215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1340.761141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1344.002797] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1351.768279] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1360.001924] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1368.326402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1368.859938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1374.022490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1375.224391] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1381.892679] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1390.545097] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1394.554691] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1406.781386] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0622h58m37s [ 1408.408192] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1411.043961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1411.067744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1411.077660] Lustre: Skipped 1 previous similar message [ 1412.133421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1412.140235] Lustre: Skipped 2 previous similar messages [ 1414.370296] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.26@tcp (stopping) [ 1416.667353] Lustre: server umount lustre-MDT0000 complete [ 1418.217479] LustreError: 50273:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1418.238674] LustreError: 50273:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1420.321544] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1421.524347] LustreError: 50269:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1424.374901] LustreError: 50268:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1424.395694] LustreError: 50268:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1426.880845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.035804] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1427.231921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.287306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1428.269715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1431.231654] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1434.600409] Lustre: DEBUG MARKER: Wait for df (36689) ... [ 1436.073495] Lustre: DEBUG MARKER: done [ 1441.761909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1441.779525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1441.800519] Lustre: Skipped 2 previous similar messages [ 1441.810266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1441.817883] Lustre: Skipped 1 previous similar message [ 1442.787314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1442.799206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.705614] Lustre: server umount lustre-OST0000 complete [ 1454.053597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1454.054875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1454.071623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1454.071632] Lustre: Skipped 3 previous similar messages [ 1456.585616] Lustre: server umount lustre-MDT0000 complete [ 1459.777946] LustreError: 50254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749972 with bad export cookie 13213547169678752439 [ 1459.784012] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1459.787281] LustreError: 50254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1460.089199] Lustre: server umount lustre-MDT0001 complete [ 1472.804300] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 1477.581031] Key type lgssc unregistered [ 1477.986414] LNet: 55193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1478.009542] LNetError: 55193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1478.041651] LNet: Removed LNI 192.168.202.126@tcp [ 1479.366299] Key type .llcrypt unregistered [ 1479.368621] Key type ._llcrypt unregistered [ 1518.276404] Key type ._llcrypt registered [ 1518.279864] Key type .llcrypt registered [ 1518.433122] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 1519.941738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1520.332404] alg: No test for adler32 (adler32-zlib) [ 1521.543889] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 1521.851885] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 1523.687295] Key type lgssc registered [ 1525.112125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1533.893409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1533.928769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.295377] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1535.319453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1535.424726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.966642] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1549.123666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.360673] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1549.396710] Lustre: 57394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1549.752719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1554.445483] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1561.911843] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1567.896720] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1575.429241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1575.630805] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1575.850784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1581.518054] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1583.738772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1587.707501] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1594.300644] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0623h01m45s [ 1596.165901] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1603.349867] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1606.076352] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1606.162070] LustreError: 57401:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1626.365702] Lustre: lustre-MDT0000: Client 64b44f68-08f2-417c-ba47-77ba50ce7ed3 (at 192.168.202.26@tcp) reconnecting [ 1646.183133] LustreError: 57401:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1648.390707] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1650.719118] Lustre: DEBUG MARKER: done [ 1658.189762] Lustre: server umount lustre-OST0000 complete [ 1662.432372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1662.437887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.457365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1663.522318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1663.533606] Lustre: Skipped 2 previous similar messages [ 1669.607458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1669.610331] Lustre: Skipped 1 previous similar message [ 1671.746400] Lustre: server umount lustre-MDT0000 complete [ 1674.969934] LustreError: 58158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750187 with bad export cookie 14440122237477493922 [ 1674.976101] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1685.655845] Lustre: server umount lustre-MDT0001 complete [ 1696.993488] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 1699.994835] Key type lgssc unregistered [ 1700.413625] LNet: 61739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.432047] LNetError: 61739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.470817] LNet: Removed LNI 192.168.202.126@tcp [ 1701.539180] Key type .llcrypt unregistered [ 1701.541272] Key type ._llcrypt unregistered [ 1752.158314] Key type ._llcrypt registered [ 1752.163235] Key type .llcrypt registered [ 1752.375917] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 1754.246491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1754.333447] alg: No test for adler32 (adler32-zlib) [ 1755.467347] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 1755.769871] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 1757.463232] Key type lgssc registered [ 1758.914746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1762.484913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1762.515528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1763.860460] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1763.884612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1764.021743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.014667] Lustre: Failing over lustre-MDT0000 [ 1765.677394] Lustre: server umount lustre-MDT0000 complete [ 1772.384977] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:04:43 (1788750283) [ 1779.028065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.516654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.158628] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1792.614034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1792.720615] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1792.757802] Lustre: 64555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1793.048188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1797.868431] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1804.886404] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1810.462863] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1817.964508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1818.255794] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1818.507947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.374914] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1828.525047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1833.555729] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1839.924616] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1847.553851] Lustre: server umount lustre-MDT0000 complete [ 1849.330208] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.336169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1849.345726] Lustre: Skipped 1 previous similar message [ 1850.918331] LustreError: 66411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750363 with bad export cookie 2855884225499086663 [ 1850.924389] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1850.931733] LustreError: 66411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1851.199690] Lustre: server umount lustre-MDT0001 complete [ 1854.454824] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1858.232584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1865.729120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.623708] Lustre: 63481:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788750366/real 1788750366] req@00000000e4ea9f77 x1875640600009600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788750382 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1870.654883] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.670229] Lustre: Skipped 1 previous similar message [ 1874.911208] Lustre: 63480:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788750371/real 1788750371] req@000000009eea9c7f x1875640600009856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788750387 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1874.953448] Lustre: 63480:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1875.807946] LustreError: 63478:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009629f9a7 x1875640600010752/t0(0) o250->MGC192.168.202.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1876.225117] LustreError: 66404:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1876.258259] LustreError: 66404:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 1876.332768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.446279] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1877.344384] Lustre: 63482:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788750373/real 1788750373] req@000000002cb61af8 x1875640600010112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788750389 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1880.420300] LustreError: 68327:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1880.441764] LustreError: 68327:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1881.089596] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1882.463751] Lustre: 63482:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788750378/real 1788750378] req@0000000078e54c38 x1875640600010240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788750394 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1885.675796] LustreError: 68354:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.713190] LustreError: 68354:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1886.691350] Lustre: 63480:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788750382/real 1788750382] req@0000000002ac1c2b x1875640600010496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788750398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1889.489171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.039077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1891.054184] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1895.419444] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1903.877920] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1909.426994] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1915.777970] Lustre: server umount lustre-MDT0000 complete [ 1916.905966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1916.914038] LustreError: 68326:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.923093] Lustre: Skipped 2 previous similar messages [ 1916.960667] LustreError: 68326:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1919.277728] LustreError: 64549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750431 with bad export cookie 2855884225499088938 [ 1919.282965] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1919.287016] LustreError: 64549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1919.582572] Lustre: server umount lustre-MDT0001 complete [ 1925.700350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1935.738545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.399419] Lustre: 63480:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788750434/real 1788750434] req@00000000344fe458 x1875640600036736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788750450 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1938.418597] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1944.626159] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x27a223926c83cd8d [ 1944.634989] Lustre: MGC192.168.202.126@tcp: Connection restored to 0@lo (at 0@lo) [ 1944.639021] Lustre: Skipped 1 previous similar message [ 1944.931013] LustreError: 66405:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1944.989505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1945.036858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1949.716462] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1953.251102] LustreError: 71009:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.276904] LustreError: 71009:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 1957.351836] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1959.141840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.405959] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 1965.036718] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1972.689640] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1979.320278] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1990.632203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1990.634239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1990.645382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1990.650808] Lustre: Skipped 1 previous similar message [ 1995.745914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1995.761876] Lustre: Skipped 4 previous similar messages [ 2000.352785] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2000.618104] Lustre: server umount lustre-MDT0000 complete [ 2000.874429] LustreError: 71008:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2000.903223] LustreError: 71008:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 2004.297555] LustreError: 66411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750516 with bad export cookie 2855884225499090317 [ 2004.306131] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.315094] LustreError: 66411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2004.697030] Lustre: server umount lustre-MDT0001 complete [ 2011.620508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2015.785196] Lustre: DEBUG MARKER: files compared the same [ 2027.685512] Lustre: server umount lustre-OST0000 complete [ 2042.920354] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 2045.693035] Key type lgssc unregistered [ 2045.975338] LNet: 74284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.996686] LNetError: 74284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.015609] LNet: Removed LNI 192.168.202.126@tcp [ 2046.955978] Key type .llcrypt unregistered [ 2046.958797] Key type ._llcrypt unregistered [ 2074.154436] Key type ._llcrypt registered [ 2074.160906] Key type .llcrypt registered [ 2074.330596] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 2075.449861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2075.530099] alg: No test for adler32 (adler32-zlib) [ 2076.660958] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 2077.099776] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 2078.848207] Key type lgssc registered [ 2080.176087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2087.491474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2087.511368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2089.222502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2092.712267] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug +malloc all [ 2099.680559] LustreError: 75699:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.689382] LustreError: 75699:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2100.329608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.656724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2106.099235] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug +malloc all [ 2112.623674] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2118.415048] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2124.968748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2125.291208] LustreError: 77519:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2125.796611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2131.668659] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug +malloc all [ 2134.014664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2140.756753] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2151.392104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2151.404338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.424167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2154.467063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2154.469184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2154.501205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.585275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2159.595738] Lustre: Skipped 2 previous similar messages [ 2164.191279] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2164.344201] Lustre: server umount lustre-OST0000 complete [ 2169.825782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.834976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2169.838671] Lustre: Skipped 1 previous similar message [ 2174.809399] Lustre: server umount lustre-MDT0000 complete [ 2174.944658] LustreError: 75696:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2174.970913] LustreError: 75696:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2178.279374] LustreError: 75680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750690 with bad export cookie 3219723854009530134 [ 2178.287428] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2178.292361] LustreError: 75680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2178.608330] Lustre: server umount lustre-MDT0001 complete [ 2190.101562] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 2193.036743] Key type lgssc unregistered [ 2193.322098] LNet: 79232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.329868] LNetError: 79232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.341460] LNet: Removed LNI 192.168.202.126@tcp [ 2194.286984] Key type .llcrypt unregistered [ 2194.290514] Key type ._llcrypt unregistered [ 2219.734407] Key type ._llcrypt registered [ 2219.738035] Key type .llcrypt registered [ 2219.880401] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 2221.346586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2221.584882] alg: No test for adler32 (adler32-zlib) [ 2222.741833] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 2222.981220] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 2224.648372] Key type lgssc registered [ 2225.850715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2234.293758] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2234.314237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2249.823373] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2254.943313] LustreError: 80618:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.126@tcp: failed processing log, type 4: rc = -110 [ 2281.631165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.827472] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2297.053807] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2305.939626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2306.059698] Lustre: *** cfs_fail_loc=706, val=0*** [ 2306.653225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2306.754253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2311.671834] LustreError: 80644:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2311.700672] LustreError: 80644:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2312.488831] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2315.850818] LustreError: 81706:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.865432] LustreError: 81706:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2320.876813] LustreError: 81705:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.886561] LustreError: 81705:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2323.387792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.751938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2329.683719] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2338.480898] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2344.607036] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2347.492974] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2347.499411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2347.524518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2349.541753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2349.542477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2349.558350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.662121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2354.668184] Lustre: Skipped 1 previous similar message [ 2359.800926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2359.813947] Lustre: Skipped 1 previous similar message [ 2361.312469] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2361.484065] Lustre: server umount lustre-OST0000 complete [ 2370.022352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2370.025278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.052225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2370.054547] Lustre: Skipped 2 previous similar messages [ 2371.866203] Lustre: server umount lustre-MDT0000 complete [ 2374.778353] LustreError: 80625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750887 with bad export cookie 16440009147567557301 [ 2374.786578] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2375.188156] Lustre: server umount lustre-MDT0001 complete [ 2385.018498] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 2387.375915] Key type lgssc unregistered [ 2387.630469] LNet: 84233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2387.635878] LNetError: 84233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2387.660616] LNet: Removed LNI 192.168.202.126@tcp [ 2388.640423] Key type .llcrypt unregistered [ 2388.646795] Key type ._llcrypt unregistered [ 2412.973316] Key type ._llcrypt registered [ 2412.975440] Key type .llcrypt registered [ 2413.124455] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 2414.998453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2415.018905] alg: No test for adler32 (adler32-zlib) [ 2416.066994] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 2416.309337] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 2418.000952] Key type lgssc registered [ 2418.987810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2426.378447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2426.396350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.546242] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2440.038080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2440.555134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2444.334397] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2445.795023] LustreError: 86332:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2445.813297] LustreError: 86332:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 2449.894240] LustreError: 86336:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.920658] LustreError: 86336:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2452.237939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2452.710875] LustreError: 86332:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.742878] LustreError: 86332:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2452.798152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2458.098896] LustreError: 86330:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.112222] LustreError: 86330:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 2459.611220] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2463.202392] LustreError: 86335:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.216749] LustreError: 86335:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2467.295949] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2467.305392] Lustre: MGS: Client aec64d53-cb58-403c-bed7-d8e8ce0f97a5 (at 0@lo) reconnecting [ 2467.331591] LustreError: 86346:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2467.598035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.699854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2472.117708] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2488.292906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2488.303115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.350454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2490.862446] Lustre: server umount lustre-OST0000 complete [ 2498.528839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2498.533245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.535129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2498.547574] Lustre: Skipped 2 previous similar messages [ 2498.565167] Lustre: Skipped 2 previous similar messages [ 2503.648341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2503.659137] Lustre: Skipped 1 previous similar message [ 2508.771110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2508.789655] Lustre: Skipped 1 previous similar message [ 2509.279135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2509.509774] Lustre: server umount lustre-MDT0000 complete [ 2513.891568] LustreError: 86336:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2513.900484] LustreError: 86336:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 2514.478441] Lustre: server umount lustre-MDT0001 complete [ 2518.228312] Lustre: server umount MGS complete [ 2529.517888] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 23:17:20 (1788751040) [ 2557.508896] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 2567.239385] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 2600.730197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.470962] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2614.088055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2614.358819] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2614.409844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2614.482676] Lustre: lustre-MDT0001: new disk, initializing [ 2614.549529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2614.571355] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2614.579942] Lustre: 93501:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2614.598977] LustreError: 93501:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2619.221276] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2628.836416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2628.982408] Lustre: lustre-OST0000: new disk, initializing [ 2628.986276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2628.993350] Lustre: 94267:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2629.083258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.696538] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2636.908756] Lustre: 94624:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000b3ba597 x1875641293119616/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788751165 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2644.689569] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2651.054907] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2651.065699] Lustre: MGS: Client fa5d6d3d-5806-4fd8-974e-c9e6125c002f (at 0@lo) reconnecting [ 2651.107135] Lustre: 92769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2651.132664] LustreError: 93487:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2651.152872] LustreError: 93487:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2651.178655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2651.284847] Lustre: lustre-MDT0000: new disk, initializing [ 2651.460528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.496107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2651.533689] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2652.577402] Lustre: 95335:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004dee1d28 x1875641293126144/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788751180 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2655.514829] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2657.650763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2657.667071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2657.728093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2679.776253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2679.781596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.788453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2683.381054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.386235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2688.485299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2688.498978] Lustre: Skipped 3 previous similar messages [ 2694.111167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2694.250651] Lustre: server umount lustre-OST0000 complete [ 2698.720055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2698.725165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.731865] Lustre: Skipped 1 previous similar message [ 2704.109851] Lustre: server umount lustre-MDT0000 complete [ 2722.271487] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2722.595566] Lustre: server umount lustre-MDT0001 complete [ 2726.274776] Lustre: server umount MGS complete [ 2738.558592] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 23:20:49 (1788751249) [ 2755.558909] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 2758.217527] Key type lgssc unregistered [ 2758.444753] LNet: 97834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2758.451059] LNetError: 97834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2758.466262] LNet: Removed LNI 192.168.202.126@tcp [ 2759.357298] Key type .llcrypt unregistered [ 2759.360758] Key type ._llcrypt unregistered [ 2778.937531] Key type ._llcrypt registered [ 2778.939621] Key type .llcrypt registered [ 2779.042641] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 2779.988042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2780.186554] alg: No test for adler32 (adler32-zlib) [ 2781.455613] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 2781.740545] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 2783.479555] Key type lgssc registered [ 2785.024928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2792.811847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2792.841893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.353557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2798.940150] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2804.706967] LustreError: 99052:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.728299] LustreError: 99052:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2806.838231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.284505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2812.305516] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2820.210514] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2826.085510] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2833.105318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2833.479575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2839.612470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2840.392410] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2847.671155] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2860.000742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2860.010144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.035107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2862.050305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2862.054486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2862.083739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2864.464141] Lustre: server umount lustre-OST0000 complete [ 2868.706542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2868.712264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.721173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2868.723784] Lustre: Skipped 1 previous similar message [ 2872.288103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2872.289824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2874.733299] Lustre: server umount lustre-MDT0000 complete [ 2877.409283] LustreError: 101541:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.427174] LustreError: 101541:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2878.811750] LustreError: 99033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751391 with bad export cookie 12430679295553595692 [ 2878.817553] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2878.830648] LustreError: 99033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2879.148689] Lustre: server umount lustre-MDT0001 complete [ 2888.211183] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 2890.899257] Key type lgssc unregistered [ 2891.226899] LNet: 102582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2891.231868] LNetError: 102582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2891.242720] LNet: Removed LNI 192.168.202.126@tcp [ 2892.157729] Key type .llcrypt unregistered [ 2892.159587] Key type ._llcrypt unregistered [ 2907.479172] Key type ._llcrypt registered [ 2907.486164] Key type .llcrypt registered [ 2907.586393] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 2908.469534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2908.553763] alg: No test for adler32 (adler32-zlib) [ 2909.540468] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 2909.717728] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 2911.391330] Key type lgssc registered [ 2912.547866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2918.400115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2918.412114] LustreError: 103650:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2918.455089] LustreError: 103660:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2918.466188] LustreError: 103650:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4964 [ 2918.478650] LustreError: 103650:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2918.484782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2918.486092] LustreError: 103650:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2920.064467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2927.191262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.516116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2932.025118] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2937.753434] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing lsmod [ 2940.520512] LustreError: 104874:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2940.521406] LustreError: 104875:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2940.531587] LustreError: 104874:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 2940.544861] LustreError: 104874:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2940.552813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2940.560208] LustreError: 104874:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2941.015971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2943.966218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.969279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2943.974707] Lustre: Skipped 1 previous similar message [ 2943.984140] Lustre: Skipped 1 previous similar message [ 2949.089292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2952.164121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2952.174547] Lustre: Skipped 4 previous similar messages [ 2957.285683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2957.290546] Lustre: Skipped 2 previous similar messages [ 2958.303609] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2958.495039] Lustre: server umount lustre-MDT0000 complete [ 2961.587090] LustreError: 103673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751473 with bad export cookie 1486252968062405655 [ 2961.592654] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2962.400194] LustreError: 103692:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2962.400412] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2962.422133] LustreError: 103692:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2962.426957] Lustre: Skipped 1 previous similar message [ 2962.438662] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2962.527293] LustreError: 104128:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000734d61d2 x1875641810047744/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2962.661474] Lustre: server umount lustre-MDT0001 complete [ 2968.058362] Lustre: server umount lustre-OST0000 complete [ 2974.592478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.892564] LustreError: 105939:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2974.970149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2978.955224] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2980.321328] LustreError: 105940:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.447676] LustreError: 105939:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2986.987995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2987.343413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2991.401971] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 2997.875815] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3002.413992] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3007.374335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3007.635818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3012.008943] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3018.137264] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3019.953642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3028.449465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3028.463056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.496378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3030.496583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3030.504868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.368883] Lustre: server umount lustre-OST0000 complete [ 3038.689211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3038.701669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3044.260532] Lustre: server umount lustre-MDT0000 complete [ 3045.858623] LustreError: 105935:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3045.878256] LustreError: 105935:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3047.751178] LustreError: 105921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751559 with bad export cookie 1486252968062406341 [ 3047.755661] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3047.760402] LustreError: 105921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3048.092835] Lustre: server umount lustre-MDT0001 complete [ 3057.644213] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 3060.089636] Key type lgssc unregistered [ 3060.361696] LNet: 109472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3060.368612] LNetError: 109472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3060.380843] LNet: Removed LNI 192.168.202.126@tcp [ 3061.117246] Key type .llcrypt unregistered [ 3061.119366] Key type ._llcrypt unregistered [ 3081.775751] Key type ._llcrypt registered [ 3081.778266] Key type .llcrypt registered [ 3081.853986] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 3082.764905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3082.830235] alg: No test for adler32 (adler32-zlib) [ 3083.809937] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 3084.038713] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 3085.703236] Key type lgssc registered [ 3086.756948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3092.615666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3092.631793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3094.068536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.897133] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3104.236730] LustreError: 110885:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.252506] LustreError: 110885:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3105.448872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.830744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3110.592455] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3117.297459] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3121.985981] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3128.265743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3128.502432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3133.985201] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3139.767200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3140.215878] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3144.640299] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3147.390903] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3151.331780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3151.340054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3151.350345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3155.424399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3155.432023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3155.438503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3157.164775] Lustre: server umount lustre-OST0000 complete [ 3160.547046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3160.559334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3160.562766] Lustre: Skipped 2 previous similar messages [ 3165.667132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3165.672405] Lustre: Skipped 1 previous similar message [ 3166.679338] Lustre: server umount lustre-MDT0000 complete [ 3169.620093] LustreError: 113415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751681 with bad export cookie 7240275697171902818 [ 3169.623496] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3169.627643] LustreError: 113415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3169.836168] Lustre: server umount lustre-MDT0001 complete [ 3178.309583] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 3180.359695] Key type lgssc unregistered [ 3180.602822] LNet: 114519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3180.611573] LNetError: 114519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3181.670879] LNet: Removed LNI 192.168.202.126@tcp [ 3182.456161] Key type .llcrypt unregistered [ 3182.458034] Key type ._llcrypt unregistered [ 3195.579317] Key type ._llcrypt registered [ 3195.582561] Key type .llcrypt registered [ 3195.669429] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 3196.511528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3196.536515] alg: No test for adler32 (adler32-zlib) [ 3197.532158] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 3197.712273] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 3199.343566] Key type lgssc registered [ 3200.182695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3206.405316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3206.417061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3207.932246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3212.143423] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3218.402908] LustreError: 115716:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3218.421058] LustreError: 115716:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3218.994730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.312291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3222.970604] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3229.167978] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3233.924183] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3239.182068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3239.480323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3245.048847] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3251.908212] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3253.813920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3269.092369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3269.093605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3269.105782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3269.119435] Lustre: Skipped 1 previous similar message [ 3272.039987] Lustre: server umount lustre-OST0000 complete [ 3275.747186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3275.755429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3275.765886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3275.773324] Lustre: Skipped 1 previous similar message [ 3279.330398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3279.332210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3281.670203] Lustre: server umount lustre-MDT0000 complete [ 3284.393619] LustreError: 115697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751796 with bad export cookie 13749135618023232564 [ 3284.411887] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3284.733948] Lustre: server umount lustre-MDT0001 complete [ 3293.236838] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 3295.499721] Key type lgssc unregistered [ 3295.774492] LNet: 119265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3295.780339] LNetError: 119265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3295.790913] LNet: Removed LNI 192.168.202.126@tcp [ 3296.373532] Key type .llcrypt unregistered [ 3296.376972] Key type ._llcrypt unregistered [ 3317.355401] Key type ._llcrypt registered [ 3317.356897] Key type .llcrypt registered [ 3317.447291] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 3318.408843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3318.586141] alg: No test for adler32 (adler32-zlib) [ 3319.754127] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 3320.028661] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 3321.725396] Key type lgssc registered [ 3322.664797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3328.603927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3328.617648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3329.952622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3333.570920] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3339.353761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3339.577333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3343.083200] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3348.596481] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3352.644597] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3358.251067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3358.625696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3364.306630] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3370.047648] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3375.027024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3381.038839] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3384.037260] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3384.040809] Lustre: Skipped 2 previous similar messages [ 3387.521421] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3387.527883] Lustre: Skipped 2 previous similar messages [ 3389.486729] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3389.516523] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3397.769616] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.26@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3397.774556] Lustre: Skipped 2 previous similar messages [ 3398.941603] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.26@tcp 192.168.0.[2,10]@tcp [ 3406.327921] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.26/24@tcp in log lustre-MDT0000 [ 3406.338114] Lustre: Skipped 2 previous similar messages [ 3407.675788] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.26/24@tcp [ 3407.685664] Lustre: Skipped 1 previous similar message [ 3419.113712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3419.114314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.118914] LustreError: Skipped 3 previous similar messages [ 3419.136745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3421.155064] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3421.156370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3421.161651] LustreError: Skipped 1 previous similar message [ 3421.171084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.229697] Lustre: server umount lustre-OST0000 complete [ 3426.279599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.288651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3426.296398] Lustre: Skipped 1 previous similar message [ 3426.315597] Lustre: Skipped 2 previous similar messages [ 3431.396152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3431.406561] Lustre: Skipped 1 previous similar message [ 3432.537165] Lustre: server umount lustre-MDT0000 complete [ 3435.549287] LustreError: 120657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751947 with bad export cookie 10348919521177357881 [ 3435.555364] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3435.933968] Lustre: server umount lustre-MDT0001 complete [ 3444.595127] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 3446.688492] Key type lgssc unregistered [ 3446.899618] LNet: 125373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3446.909939] LNetError: 125373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3446.927437] LNet: Removed LNI 192.168.202.126@tcp [ 3447.586634] Key type .llcrypt unregistered [ 3447.588746] Key type ._llcrypt unregistered [ 3469.008874] Key type ._llcrypt registered [ 3469.011233] Key type .llcrypt registered [ 3469.083153] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 3470.142196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3470.163906] alg: No test for adler32 (adler32-zlib) [ 3471.175779] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 3471.418928] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 3473.025089] Key type lgssc registered [ 3474.084496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3480.300865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3480.322863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3481.671694] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3481.692485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3481.698622] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.26/24@tcp [ 3485.015804] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3490.789759] LustreError: 126909:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3490.799217] LustreError: 126909:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3490.879198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.113992] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3491.137740] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.26/24@tcp [ 3491.181393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3494.407868] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3499.860276] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3504.538917] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3509.568415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3509.890286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3515.064305] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3520.568833] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3526.339107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3531.747252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3531.747608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3531.757380] LustreError: Skipped 1 previous similar message [ 3531.758275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3531.768353] Lustre: Skipped 1 previous similar message [ 3536.870991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3536.881292] Lustre: Skipped 1 previous similar message [ 3541.997044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3542.000292] Lustre: Skipped 1 previous similar message [ 3546.079395] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3546.246225] Lustre: server umount lustre-OST0000 complete [ 3552.224821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3552.226901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3552.247668] Lustre: Skipped 1 previous similar message [ 3552.256865] Lustre: Skipped 1 previous similar message [ 3555.070286] Lustre: server umount lustre-MDT0000 complete [ 3557.349501] LustreError: 126904:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3557.359563] LustreError: 126904:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3557.910983] LustreError: 126889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752070 with bad export cookie 3580366268971728372 [ 3557.912893] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3557.917672] LustreError: 126889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3558.152902] Lustre: server umount lustre-MDT0001 complete [ 3565.937084] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 3568.079831] Key type lgssc unregistered [ 3568.351655] LNet: 130494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3568.359596] LNetError: 130494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3568.385725] LNet: Removed LNI 192.168.202.126@tcp [ 3569.054582] Key type .llcrypt unregistered [ 3569.055534] Key type ._llcrypt unregistered [ 3586.919422] Key type ._llcrypt registered [ 3586.924185] Key type .llcrypt registered [ 3587.011519] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 3597.034550] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 3597.896970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3597.916337] alg: No test for adler32 (adler32-zlib) [ 3598.882923] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 3599.038027] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 3600.663203] Key type lgssc registered [ 3601.520754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3626.340553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3626.380490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.505374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3627.523324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3627.583873] Lustre: lustre-MDT0000: new disk, initializing [ 3627.637139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3627.646409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3630.899554] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3639.027393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3639.085472] Lustre: 133867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3639.120542] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3639.123722] Lustre: Skipped 1 previous similar message [ 3639.165937] Lustre: lustre-MDT0001: new disk, initializing [ 3639.231057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3639.260389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3639.268574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3642.245376] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3649.385431] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3653.953639] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3659.129858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3659.277321] Lustre: lustre-OST0000: new disk, initializing [ 3659.279360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3659.284364] Lustre: 135832:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3659.334342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3663.567139] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3665.976390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3665.986057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3666.101585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3670.639153] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3673.956172] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3674.106863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3677.457676] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3677.601347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3681.771091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3681.783324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3681.785332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3681.792969] LustreError: Skipped 1 previous similar message [ 3681.801569] Lustre: Skipped 1 previous similar message [ 3685.111880] Lustre: server umount lustre-OST0000 complete [ 3690.464727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3690.474247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3690.490531] Lustre: Skipped 1 previous similar message [ 3690.494661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3692.007133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3692.010086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3692.026070] Lustre: Skipped 1 previous similar message [ 3694.017201] Lustre: server umount lustre-MDT0000 complete [ 3696.796414] LustreError: 133860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752209 with bad export cookie 18326605254874785099 [ 3696.801405] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3696.805393] LustreError: 133860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3697.042905] Lustre: server umount lustre-MDT0001 complete [ 3702.785151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3703.147447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3706.507557] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3708.389525] LustreError: 138049:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3708.403592] LustreError: 138049:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3712.851965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3713.072524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3716.141940] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3721.463626] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3725.335635] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3730.341469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3730.549886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3734.753694] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3739.549521] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3742.703844] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3749.171211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 3752.038713] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3752.209330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3758.175721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3758.323305] Lustre: lustre-OST0001: new disk, initializing [ 3758.328477] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3758.335691] Lustre: 141461:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3758.416249] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3763.075834] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3765.324187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3765.333402] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3765.393500] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3769.673692] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3769.794359] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3772.808478] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3772.948867] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3776.889800] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f23c450f800.ost_server_uuid 50 [ 3777.948898] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f23c450f800.ost_server_uuid in FULL state after 0 sec [ 3784.854425] Lustre: server umount lustre-OST0001 complete [ 3786.209044] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3786.220269] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.224176] LustreError: 139867:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3786.224844] Lustre: Skipped 1 previous similar message [ 3790.307332] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3790.314984] LustreError: Skipped 1 previous similar message [ 3790.319960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3790.330405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3790.334769] Lustre: Skipped 1 previous similar message [ 3791.342846] LustreError: 139867:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.358659] LustreError: 139867:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3793.465844] Lustre: server umount lustre-OST0000 complete [ 3796.452341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3796.459485] Lustre: Skipped 3 previous similar messages [ 3806.689728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3806.699583] Lustre: Skipped 3 previous similar messages [ 3810.274266] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3810.412602] Lustre: server umount lustre-MDT0000 complete [ 3811.812480] LustreError: 138049:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3813.083603] LustreError: 138031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752325 with bad export cookie 18326605254874785981 [ 3813.088317] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3813.091858] LustreError: 138031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3813.389314] Lustre: server umount lustre-MDT0001 complete [ 3827.104686] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 3828.999761] Key type lgssc unregistered [ 3829.233746] LNet: 144460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3829.240248] LNetError: 144460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3830.313156] LNet: Removed LNI 192.168.202.126@tcp [ 3830.946443] Key type .llcrypt unregistered [ 3830.948931] Key type ._llcrypt unregistered [ 3858.108526] Key type ._llcrypt registered [ 3858.110485] Key type .llcrypt registered [ 3858.178931] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 3867.540716] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 3868.325984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3868.375040] alg: No test for adler32 (adler32-zlib) [ 3869.317676] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 3869.561168] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 3871.207210] Key type lgssc registered [ 3872.077784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3895.685500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3895.708971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3896.946957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3896.971532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3897.079374] Lustre: lustre-MDT0000: new disk, initializing [ 3897.175907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3897.195377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3900.558409] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3908.067308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3908.133851] Lustre: 148587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3908.171985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3908.174926] Lustre: Skipped 1 previous similar message [ 3908.224354] Lustre: lustre-MDT0001: new disk, initializing [ 3908.273759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3908.306996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3908.324391] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3911.390559] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3918.326610] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3922.224916] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3927.536598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3927.737501] Lustre: lustre-OST0000: new disk, initializing [ 3927.740496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3927.744782] Lustre: 150552:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3927.822590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3932.117820] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3934.753918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3934.763092] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3934.838327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3938.660340] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3942.619302] Lustre: Failing over lustre-OST0000 [ 3942.729276] Lustre: server umount lustre-OST0000 complete [ 3957.126843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3957.291854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3957.297606] Lustre: Skipped 1 previous similar message [ 3957.307413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3957.316798] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3960.799101] Lustre: 146810:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788752457/real 1788752457] req@0000000059846036 x1875642816690560/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788752473 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3960.815780] Lustre: 146810:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3960.963229] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3961.375233] Lustre: 146812:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788752457/real 1788752457] req@0000000095a6fcb8 x1875642816690304/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788752473 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3961.398281] Lustre: 146812:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3961.580315] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3962.364040] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3962.368239] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3963.417169] Lustre: Failing over lustre-MDT0000 [ 3963.552566] Lustre: server umount lustre-MDT0000 complete [ 3964.898944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3964.903357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.495260] Lustre: 146811:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788752462/real 1788752462] req@00000000b4d2b5ed x1875642816691584/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788752478 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3966.519144] Lustre: 146811:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3967.459301] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3967.473878] Lustre: Skipped 1 previous similar message [ 3971.551205] Lustre: 146812:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788752467/real 1788752467] req@00000000726f5dc1 x1875642816692736/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788752483 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3971.792213] LustreError: 148593:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3971.813572] LustreError: 148593:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 3972.577792] LustreError: 148594:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3972.587913] LustreError: 148594:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3976.912033] LustreError: 148592:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3979.174766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3979.222322] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3979.227278] Lustre: 148590:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.126@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3979.392375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3979.430055] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3982.035430] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3982.290891] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 3984.873889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3984.878139] Lustre: Skipped 1 previous similar message [ 3984.892886] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3984.925511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3987.937587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3987.944090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3987.959309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3989.986568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3993.096906] Lustre: server umount lustre-OST0000 complete [ 4000.224604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4000.228028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.232651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4000.241084] Lustre: Skipped 2 previous similar messages [ 4000.256176] Lustre: Skipped 2 previous similar messages [ 4002.131341] Lustre: server umount lustre-MDT0000 complete [ 4004.689046] LustreError: 149389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752516 with bad export cookie 923726922626043428 [ 4004.701271] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4004.917524] Lustre: server umount lustre-MDT0001 complete [ 4011.078762] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 4012.778380] Key type lgssc unregistered [ 4013.033428] LNet: 154199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4013.040449] LNetError: 154199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4013.053730] LNet: Removed LNI 192.168.202.126@tcp [ 4013.683057] Key type .llcrypt unregistered [ 4013.684509] Key type ._llcrypt unregistered [ 4042.488218] Key type ._llcrypt registered [ 4042.491238] Key type .llcrypt registered [ 4042.593977] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 4043.687234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4043.707737] alg: No test for adler32 (adler32-zlib) [ 4044.711341] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 4044.876048] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 4046.527377] Key type lgssc registered [ 4047.440299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4052.864379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4052.873577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4054.140181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4059.173735] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4064.750602] LustreError: 155719:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.773353] LustreError: 155719:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 4069.872505] LustreError: 155738:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4072.212802] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4074.976904] LustreError: 155719:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4076.611443] LustreError: 155749:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4076.622110] LustreError: 155749:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4076.628586] LustreError: 155749:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 4077.201714] Lustre: server umount lustre-MDT0000 complete [ 4103.278294] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 4121.439609] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 4162.543608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4162.995331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4163.036089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4163.140116] Lustre: lustre-MDT0000: new disk, initializing [ 4163.355278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4163.406881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4168.659668] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4180.406612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4180.579336] Lustre: 159079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4180.655642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4180.659319] Lustre: Skipped 1 previous similar message [ 4180.728652] Lustre: lustre-MDT0001: new disk, initializing [ 4180.841637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4180.898994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4180.915741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4186.108964] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4195.293041] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4201.132747] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4207.327578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4207.564504] Lustre: lustre-OST0000: new disk, initializing [ 4207.571327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4207.580119] Lustre: 161047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4207.735851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4213.824614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4213.837201] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4213.962685] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4213.979918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4223.571976] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4227.318417] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4229.030981] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 23:45:40 (1788752740) [ 4447.204859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4447.210591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4447.235160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4449.761783] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.763219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4449.779361] Lustre: Skipped 1 previous similar message [ 4449.804522] Lustre: Skipped 2 previous similar messages [ 4451.586101] Lustre: server umount lustre-MDT0000 complete [ 4454.884677] LustreError: 162818:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4454.904644] LustreError: 162818:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 4455.160139] LustreError: 159072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752967 with bad export cookie 6286616871894107237 [ 4455.164839] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.172699] LustreError: 159072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4460.004750] LustreError: 159089:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4460.006547] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4460.044777] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4461.522234] Lustre: server umount lustre-MDT0001 complete [ 4471.846994] Lustre: server umount lustre-OST0000 complete [ 4481.688778] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 4493.194070] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 4533.485358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4533.749362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4533.799643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4533.864100] Lustre: lustre-MDT0000: new disk, initializing [ 4533.929243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4533.944590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4539.219582] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4550.461945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4550.602468] Lustre: 165704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4550.678391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4550.683136] Lustre: Skipped 1 previous similar message [ 4550.826322] Lustre: lustre-MDT0001: new disk, initializing [ 4550.952093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4551.014565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4551.038678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4555.537919] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4565.187308] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4571.863946] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4579.709734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4580.138674] Lustre: lustre-OST0000: new disk, initializing [ 4580.142851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4580.149823] Lustre: 167674:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4580.264871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4587.533300] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4588.622797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4588.646275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4588.777548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4598.127797] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4604.172142] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4604.477970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4609.202143] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4609.435366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4614.625320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4614.636818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4614.657077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4614.667116] Lustre: Skipped 2 previous similar messages [ 4618.226512] Lustre: server umount lustre-OST0000 complete [ 4622.832806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4622.838724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4622.849299] Lustre: Skipped 1 previous similar message [ 4622.852323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4628.644200] Lustre: server umount lustre-MDT0000 complete [ 4629.993039] LustreError: 167458:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4630.012753] LustreError: 167458:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 4632.695027] LustreError: 165695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753144 with bad export cookie 6286616871894266074 [ 4632.708117] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4632.708249] LustreError: 165695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4633.286985] Lustre: server umount lustre-MDT0001 complete [ 4642.601974] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:52:33 (1788753153) [ 4659.763617] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 4670.079790] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 4721.680913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4722.121931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4722.161451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4722.268677] Lustre: lustre-MDT0000: new disk, initializing [ 4722.386535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4722.419112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4729.683546] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4741.833415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4741.988305] Lustre: 172433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4742.308951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4747.657555] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4759.801937] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4767.027612] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4775.713288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4776.452337] Lustre: lustre-OST0000: new disk, initializing [ 4776.456537] Lustre: Skipped 1 previous similar message [ 4776.463379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4776.471931] Lustre: Skipped 2 previous similar messages [ 4776.477917] Lustre: 174410:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4785.826975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4785.838759] Lustre: Skipped 1 previous similar message [ 4785.849476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4786.080935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4786.710297] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4799.024041] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4811.749844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4811.761774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4811.767834] Lustre: Skipped 1 previous similar message [ 4811.791852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4811.825477] Lustre: Skipped 3 previous similar messages [ 4813.846601] Lustre: server umount lustre-OST0000 complete [ 4818.916952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4824.844098] Lustre: server umount lustre-MDT0000 complete [ 4827.107989] LustreError: 174416:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4827.129567] LustreError: 174416:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4829.743383] LustreError: 172425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753341 with bad export cookie 6286616871894266956 [ 4829.749249] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4829.762977] LustreError: 172425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4830.499407] Lustre: server umount lustre-MDT0001 complete [ 4842.185130] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:55:52 (1788753352) [ 4864.598259] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 4878.247629] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 4918.996597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4919.341253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4919.389565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4919.508418] Lustre: lustre-MDT0000: new disk, initializing [ 4919.631327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4919.637889] Lustre: Skipped 2 previous similar messages [ 4919.657584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4924.790487] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4936.455496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4936.582658] Lustre: 178738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4936.864834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4942.185570] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4954.599969] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4961.384221] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4969.368882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4969.692505] Lustre: 180714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4977.060551] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 4979.840957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4980.021888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4988.864796] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5000.682544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5000.689066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5000.703502] LustreError: Skipped 1 previous similar message [ 5000.717499] Lustre: Skipped 3 previous similar messages [ 5000.724057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5000.762901] Lustre: Skipped 4 previous similar messages [ 5005.175624] Lustre: server umount lustre-OST0000 complete [ 5017.012584] Lustre: server umount lustre-MDT0000 complete [ 5021.152876] LustreError: 178750:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5021.178965] LustreError: 178750:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5022.187799] LustreError: 178730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753534 with bad export cookie 6286616871894268041 [ 5022.199374] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5022.211229] LustreError: 178730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5022.510982] Lustre: server umount lustre-MDT0001 complete [ 5036.540591] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 5040.007655] Key type lgssc unregistered [ 5040.327294] LNet: 182570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5040.330666] LNetError: 182570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5040.342500] LNet: Removed LNI 192.168.202.126@tcp [ 5041.157525] Key type .llcrypt unregistered [ 5041.159387] Key type ._llcrypt unregistered [ 5072.878197] Key type ._llcrypt registered [ 5072.880211] Key type .llcrypt registered [ 5073.081153] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 5074.992407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5075.018176] alg: No test for adler32 (adler32-zlib) [ 5076.198304] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 5076.446915] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 5078.135272] Key type lgssc registered [ 5079.758772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5089.704421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5089.723627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5091.214934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5097.435575] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5101.536846] LustreError: 183988:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.572084] LustreError: 183988:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5106.658867] LustreError: 183989:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.284731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5108.616063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5114.269815] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5123.116131] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5130.113223] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5137.742773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5138.206161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5145.710514] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5154.414469] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5161.772487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5184.494658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5184.521914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5184.550185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5185.353156] Lustre: server umount lustre-OST0000 complete [ 5190.625268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5190.635786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5190.647962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5192.673541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5192.674737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5192.710648] Lustre: Skipped 1 previous similar message [ 5197.797631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5197.808648] Lustre: Skipped 1 previous similar message [ 5199.944189] Lustre: server umount lustre-MDT0000 complete [ 5201.892598] LustreError: 184718:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5201.923027] LustreError: 184718:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5204.874150] LustreError: 183968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753717 with bad export cookie 10045799312279295233 [ 5204.876286] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5204.896880] LustreError: 183968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5205.451961] Lustre: server umount lustre-MDT0001 complete [ 5218.972792] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 5222.121212] Key type lgssc unregistered [ 5222.496724] LNet: 187560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5222.513535] LNetError: 187560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5223.589458] LNet: Removed LNI 192.168.202.126@tcp [ 5224.353186] Key type .llcrypt unregistered [ 5224.356023] Key type ._llcrypt unregistered [ 5249.434614] Key type ._llcrypt registered [ 5249.436789] Key type .llcrypt registered [ 5249.515374] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 5250.582132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5250.653886] alg: No test for adler32 (adler32-zlib) [ 5251.734074] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 5251.973488] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 5253.690591] Key type lgssc registered [ 5254.839629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5263.895955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5263.929117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5265.603080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5271.024478] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5275.112362] LustreError: 188976:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5275.133602] LustreError: 188976:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5280.224189] LustreError: 188975:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5281.454376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5281.973827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5287.601163] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5296.408573] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5302.923068] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5310.592618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5310.867984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5317.685514] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5325.425645] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5335.429360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5342.689047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5342.700796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5342.720096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5345.766484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5345.767820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5345.813380] Lustre: Skipped 1 previous similar message [ 5347.557586] Lustre: lustre-OST0000: Not available for connect from 192.168.202.26@tcp (stopping) [ 5349.007925] Lustre: server umount lustre-OST0000 complete [ 5357.362515] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9f23c52b8000.ost_server_uuid 50 [ 5359.275394] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5373.920606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5373.931357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5373.953519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5376.489495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5380.157867] Lustre: server umount lustre-MDT0000 complete [ 5381.609021] LustreError: 188976:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5381.624887] LustreError: 188976:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5383.896925] LustreError: 188956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753896 with bad export cookie 14203236786623524518 [ 5383.903490] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5384.175316] Lustre: server umount lustre-MDT0001 complete [ 5392.372922] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 00:05:03 (1788753903) [ 5399.691045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5400.610041] LustreError: 192917:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5400.764961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.193969] LustreError: 192918:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5406.681103] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5415.394878] LustreError: 192917:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5415.423858] LustreError: 192917:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5416.751281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5417.036785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5421.834800] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5430.286608] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5436.245749] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5443.476134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5443.838496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5444.851878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5450.809867] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5458.821615] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5466.846068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5467.274840] Lustre: lustre-OST0001: new disk, initializing [ 5467.291476] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5467.308628] Lustre: 195760:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5467.467214] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5473.458426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5473.468651] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5473.629630] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5474.791549] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5484.852416] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5488.098556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5488.116821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5488.127097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5488.129568] Lustre: Skipped 2 previous similar messages [ 5494.075685] Lustre: server umount lustre-OST0000 complete [ 5494.250368] LustreError: 194741:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5494.260616] LustreError: 194741:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5501.048120] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5501.330377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5506.935905] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5507.204909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5524.961060] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5524.970486] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5524.985254] Lustre: Skipped 1 previous similar message [ 5525.000157] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5525.007350] Lustre: Skipped 4 previous similar messages [ 5527.116669] Lustre: server umount lustre-OST0001 complete [ 5534.689282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5534.708807] LustreError: Skipped 1 previous similar message [ 5537.405719] Lustre: server umount lustre-MDT0000 complete [ 5540.322705] LustreError: 192913:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5540.338983] LustreError: 192913:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 18 previous similar messages [ 5541.807739] LustreError: 192897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754054 with bad export cookie 14203236786623525582 [ 5541.822946] LustreError: 192897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5541.829842] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5542.498775] Lustre: server umount lustre-MDT0001 complete [ 5567.745108] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 00:07:59 (1788754079) [ 5575.031979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5575.295957] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5575.382986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5575.670214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5580.684311] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5590.053606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5590.149178] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5590.188712] Lustre: 199200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5595.904178] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5603.834423] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5610.065548] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5617.211890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5617.360777] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5623.967875] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5626.055984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5631.782319] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5638.544328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5638.596664] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5645.273896] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5645.564558] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5651.982679] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5656.053960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.065980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5656.070405] Lustre: Skipped 4 previous similar messages [ 5656.097657] Lustre: Skipped 4 previous similar messages [ 5661.642361] Lustre: server umount lustre-OST0000 complete [ 5662.942266] LustreError: 201541:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.496294] Lustre: server umount lustre-OST0001 complete [ 5691.882038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5691.898198] Lustre: Skipped 6 previous similar messages [ 5695.071849] Lustre: server umount lustre-MDT0000 complete [ 5696.930054] Lustre: 188286:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788754193/real 1788754193] req@0000000029645ff0 x1875644267005568/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788754209 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5696.951138] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5696.962079] Lustre: Skipped 2 previous similar messages [ 5698.938094] LustreError: 199193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754211 with bad export cookie 14203236786623526996 [ 5698.950603] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5698.968201] LustreError: 199193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5699.445270] Lustre: server umount lustre-MDT0001 complete [ 5724.493896] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 00:10:35 (1788754235) [ 5741.722643] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 5753.171425] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 5794.071755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5794.475443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5794.500411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5794.632262] Lustre: lustre-MDT0000: new disk, initializing [ 5794.761806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5794.764797] Lustre: Skipped 3 previous similar messages [ 5794.802620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5800.109922] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5811.812034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.978412] Lustre: 207301:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5812.074777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5812.079387] Lustre: Skipped 1 previous similar message [ 5812.211602] Lustre: lustre-MDT0001: new disk, initializing [ 5812.413286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5812.437212] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5817.960344] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5828.522760] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5834.700684] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5842.267553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5842.572237] Lustre: lustre-OST0000: new disk, initializing [ 5842.582649] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5842.602616] Lustre: 209266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5849.492450] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5850.671969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5850.691459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5850.831483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5859.175504] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5865.186117] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5865.404361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5870.410498] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5870.640265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5876.707630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5876.709699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5876.728427] LustreError: Skipped 1 previous similar message [ 5876.730390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5876.743353] Lustre: Skipped 1 previous similar message [ 5876.776516] Lustre: Skipped 1 previous similar message [ 5878.839871] Lustre: server umount lustre-OST0000 complete [ 5897.183384] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5897.461890] Lustre: server umount lustre-MDT0000 complete [ 5901.217713] LustreError: 207292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754413 with bad export cookie 14203236786623528781 [ 5901.230813] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5901.232987] LustreError: 207292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5901.624572] Lustre: server umount lustre-MDT0001 complete [ 5910.422603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5910.855026] LustreError: 211487:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5910.871518] LustreError: 211487:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 10 previous similar messages [ 5916.060348] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5925.740439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5926.164917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5926.168092] Lustre: Skipped 3 previous similar messages [ 5931.353119] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5938.718661] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5943.935643] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5950.347603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5956.548473] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 5963.602343] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5968.129842] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5968.337507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5973.229165] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5973.420501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5976.547717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5976.552192] LustreError: Skipped 1 previous similar message [ 5981.647755] Lustre: server umount lustre-OST0000 complete [ 5989.021401] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5994.863468] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6024.602721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6031.928506] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6040.721214] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6041.263208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.270353] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.272145] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6041.289836] Lustre: Skipped 1 previous similar message [ 6042.430547] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6046.688569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6046.692103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6046.696398] LustreError: Skipped 1 previous similar message [ 6046.697179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6046.697186] Lustre: Skipped 9 previous similar messages [ 6046.749514] Lustre: Skipped 5 previous similar messages [ 6048.544466] Lustre: server umount lustre-OST0000 complete [ 6067.167209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6067.512868] Lustre: server umount lustre-MDT0000 complete [ 6071.285805] LustreError: 211467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754583 with bad export cookie 14203236786623529663 [ 6071.305387] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6071.312880] LustreError: 211467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6071.776867] Lustre: server umount lustre-MDT0001 complete [ 6081.776712] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 00:16:32 (1788754592) [ 6089.325791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6095.157463] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6104.861493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.476204] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6121.786078] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6128.262636] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6135.516666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6143.439447] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6152.931320] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6158.750841] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6161.311487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6167.911975] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6168.216246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6177.410775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6177.661428] Lustre: lustre-OST0001: new disk, initializing [ 6177.664307] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6177.669682] Lustre: 220844:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6186.104114] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6186.200791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6186.207669] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6186.375166] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6198.147673] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6203.191941] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6203.431754] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6209.051559] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6209.334596] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6212.073780] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6212.081454] LustreError: Skipped 2 previous similar messages [ 6218.070866] Lustre: server umount lustre-OST0001 complete [ 6222.309808] LustreError: 219276:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6222.351604] LustreError: 219276:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [ 6226.602924] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6226.818653] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6232.187016] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6232.485527] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6262.946939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6263.235873] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6263.248234] Lustre: Skipped 6 previous similar messages [ 6268.397382] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6268.411612] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6268.418388] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6268.444938] Lustre: Skipped 1 previous similar message [ 6271.138510] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6278.888748] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6280.530255] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6288.941845] Lustre: server umount lustre-OST0001 complete [ 6309.374232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6309.388124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6309.395384] Lustre: Skipped 7 previous similar messages [ 6309.417696] Lustre: Skipped 14 previous similar messages [ 6321.119152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6321.390455] Lustre: server umount lustre-OST0000 complete [ 6332.189254] Lustre: server umount lustre-MDT0000 complete [ 6336.658326] LustreError: 217435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754848 with bad export cookie 14203236786623530629 [ 6336.665697] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6336.673793] LustreError: 217435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6343.211653] Lustre: server umount lustre-MDT0001 complete [ 6376.970610] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 00:21:27 (1788754887) [ 6384.089391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6384.477392] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6384.504512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6390.973214] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6400.778053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6400.875782] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6400.896717] Lustre: 226095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6406.511891] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6416.096539] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6422.371980] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6428.426788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6428.709958] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6434.834890] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6442.158811] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6452.928686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6453.013163] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6459.739481] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6467.227936] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6468.909439] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6474.006444] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6474.165934] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6481.264362] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6481.793097] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6490.833437] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f23c35e2800.ost_server_uuid 50 [ 6493.543124] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f23c35e2800.ost_server_uuid in IDLE state after 0 sec [ 6495.506761] Lustre: Permanently deactivating lustre-OST0001 [ 6495.550260] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6500.286109] Lustre: Permanently reactivating lustre-OST0001 [ 6500.384165] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6500.392842] Lustre: Skipped 2 previous similar messages [ 6504.935619] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6504.949887] LustreError: Skipped 6 previous similar messages [ 6519.263149] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6519.413564] Lustre: server umount lustre-OST0001 complete [ 6538.207196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6538.466294] Lustre: server umount lustre-OST0000 complete [ 6549.489423] Lustre: server umount lustre-MDT0000 complete [ 6553.367597] LustreError: 226844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755065 with bad export cookie 14203236786623531917 [ 6553.381609] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6553.383278] LustreError: 226844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6553.788348] Lustre: server umount lustre-MDT0001 complete [ 6581.397644] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 00:24:52 (1788755092) [ 6591.021825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6591.420981] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6591.450778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6591.459518] Lustre: Skipped 2 previous similar messages [ 6596.606758] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6606.916834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6606.989847] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6607.004766] Lustre: 232791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6612.123717] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6620.777217] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6627.194956] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6635.637210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6635.899844] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6635.931336] Lustre: Permanently deactivating lustre-OST0000 [ 6635.947366] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6645.339133] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6645.444236] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6654.573877] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6662.175914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6662.313060] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6669.126166] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6669.808218] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6678.301566] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6680.625743] Lustre: Permanently reactivating lustre-OST0000 [ 6680.678416] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6680.693047] Lustre: Skipped 2 previous similar messages [ 6689.348494] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6689.350833] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 44 seconds. I think it's dead, and I am evicting it. exp 00000000e00db2e2, cur 1788755201 deadline 1788755187 last 1788755157 [ 6689.354528] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6689.354884] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6689.485053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6704.513130] Lustre: server umount lustre-OST0001 complete [ 6715.157905] Lustre: server umount lustre-OST0000 complete [ 6726.336520] Lustre: server umount lustre-MDT0000 complete [ 6730.826534] LustreError: 232784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755243 with bad export cookie 14203236786623533583 [ 6730.834232] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6730.841865] LustreError: 232784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6731.136696] Lustre: server umount lustre-MDT0001 complete [ 6742.655640] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 6747.095968] Key type lgssc unregistered [ 6747.338422] LNet: 237674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6747.350223] LNetError: 237674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6747.368693] LNet: Removed LNI 192.168.202.126@tcp [ 6748.161497] Key type .llcrypt unregistered [ 6748.164098] Key type ._llcrypt unregistered [ 6775.876431] Key type ._llcrypt registered [ 6775.880471] Key type .llcrypt registered [ 6776.043526] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 6778.399175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6778.457112] alg: No test for adler32 (adler32-zlib) [ 6779.804548] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 6780.164310] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 6781.951746] Key type lgssc registered [ 6783.981865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6798.197389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6798.230955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6799.974779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6806.179495] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6810.595576] LustreError: 239139:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6810.626233] LustreError: 239139:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6815.715151] LustreError: 239140:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6817.004051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6817.204451] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6817.235390] Lustre: 239128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6817.248216] Lustre: Permanently deactivating lustre-MDT0001 [ 6817.262820] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6817.623257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6823.175739] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6823.977834] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6831.915830] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6838.710160] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6846.762652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6847.336280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6855.314521] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6862.691201] LustreError: 240979:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6862.700616] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6864.085743] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6867.957482] LustreError: 240979:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6870.025699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6872.460388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6872.773365] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6878.218713] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6878.228668] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6880.783293] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 6889.172838] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6893.652572] Lustre: Permanently reactivating lustre-MDT0001 [ 6893.704824] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6893.708731] Lustre: Skipped 2 previous similar messages [ 6895.936864] Lustre: lustre-MDT0001: Client 87d88d8d-12f6-4c6d-ad04-8fd74d2d0063 (at 192.168.202.26@tcp) reconnecting [ 6903.645411] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6904.002289] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6904.010949] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6904.015098] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6904.020840] LustreError: 242902:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6904.021937] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6904.025852] LustreError: 242902:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6904.043326] LustreError: 242902:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.126@tcp: cfg command failed: rc = -108 [ 6904.049063] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6904.049063] [ 6904.057711] LustreError: 239131:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6905.488489] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6910.955346] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6911.281807] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6916.983801] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6917.281599] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6919.438979] Lustre: Permanently deactivating lustre-MDT0001 [ 6919.474644] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6919.483260] Lustre: Skipped 2 previous similar messages [ 6925.983422] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6935.536640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6935.544211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6935.560233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6939.620746] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6939.625076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6939.653798] Lustre: Skipped 1 previous similar message [ 6939.671353] Lustre: Skipped 2 previous similar messages [ 6941.558307] Lustre: server umount lustre-MDT0000 complete [ 6944.743786] LustreError: 239140:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6944.781748] LustreError: 239140:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 6945.684742] LustreError: 242689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755457 with bad export cookie 17682327285449043368 [ 6945.685830] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6945.697078] LustreError: 242689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6945.967607] Lustre: server umount lustre-MDT0001 complete [ 6953.145325] Lustre: server umount lustre-OST0000 complete [ 6967.678480] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 00:31:19 (1788755479) [ 6979.990475] Lustre: server umount lustre-OST0001 complete [ 6986.205328] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 6998.150821] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 7040.985179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7041.392739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7041.401081] Lustre: Skipped 2 previous similar messages [ 7041.463767] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7041.626901] Lustre: lustre-MDT0000: new disk, initializing [ 7041.754726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7041.770316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7046.127937] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7058.149140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7058.247761] Lustre: 247401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7058.293774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7058.297142] Lustre: Skipped 1 previous similar message [ 7058.342692] Lustre: lustre-MDT0001: new disk, initializing [ 7058.408489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7058.423349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7058.429382] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7063.966479] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7074.600575] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7081.883214] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7090.958564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7091.706052] Lustre: lustre-OST0000: new disk, initializing [ 7091.711566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7091.720230] Lustre: 249374:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7091.857245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7100.430250] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7101.081569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7101.096664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7101.291741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7111.353566] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7114.510919] LustreError: 247408:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7124.543155] LustreError: 247408:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7124.587892] LustreError: 247409:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7124.726560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7124.924670] Lustre: lustre-OST0001: new disk, initializing [ 7124.933978] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7124.940408] Lustre: 250491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7125.063795] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7131.285750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7131.311077] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7131.519198] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7132.539751] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7134.662680] LustreError: 247409:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7144.799940] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7146.766733] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7152.106912] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7152.113364] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7152.122466] LustreError: Skipped 1 previous similar message [ 7152.125334] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7152.139362] Lustre: Skipped 2 previous similar messages [ 7152.164769] Lustre: Skipped 1 previous similar message [ 7155.368113] Lustre: server umount lustre-OST0001 complete [ 7157.220703] LustreError: 249360:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7157.240783] LustreError: 249360:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7172.580536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7172.594551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7172.602940] Lustre: Skipped 1 previous similar message [ 7177.702033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7177.703608] LustreError: 249757:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7177.708829] Lustre: Skipped 1 previous similar message [ 7177.734741] LustreError: 249757:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [ 7178.948560] Lustre: server umount lustre-OST0000 complete [ 7186.400801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7186.407547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7186.418966] Lustre: Skipped 1 previous similar message [ 7186.425186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7189.665370] Lustre: server umount lustre-MDT0000 complete [ 7193.904129] LustreError: 247393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755706 with bad export cookie 17682327285449046182 [ 7193.906119] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7193.914884] LustreError: 247393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7200.312164] Lustre: server umount lustre-MDT0001 complete [ 7213.577490] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 7216.935050] Key type lgssc unregistered [ 7217.274705] LNet: 252526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7217.296731] LNetError: 252526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7217.322691] LNet: Removed LNI 192.168.202.126@tcp [ 7218.404146] Key type .llcrypt unregistered [ 7218.407080] Key type ._llcrypt unregistered [ 7264.345537] Key type ._llcrypt registered [ 7264.348570] Key type .llcrypt registered [ 7264.484186] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 7265.761517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7265.783195] alg: No test for adler32 (adler32-zlib) [ 7266.946664] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 7267.394143] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 7269.175412] Key type lgssc registered [ 7271.394971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7281.898463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7281.954225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7283.396178] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7283.460878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7283.782233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7290.503688] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7302.243854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7302.376106] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7302.440045] Lustre: 254732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7302.803904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7308.833856] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7318.807091] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7326.092985] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7333.920887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7334.133706] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7334.437869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7342.545082] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7344.300409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7351.350529] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7405.537154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7405.563385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7405.625435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7406.064949] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7406.070795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7406.132532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7406.135360] Lustre: Skipped 1 previous similar message [ 7410.147684] Lustre: server umount lustre-OST0000 complete [ 7415.880109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7431.600182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7431.895843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7439.255688] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7446.248657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7446.271876] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7448.797807] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7456.556556] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7456.588178] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7492.577938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7492.581294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7492.593842] LustreError: Skipped 1 previous similar message [ 7492.594960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7492.626926] Lustre: Skipped 1 previous similar message [ 7497.612892] Lustre: server umount lustre-OST0000 complete [ 7502.305922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7502.315295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7502.341297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7502.354824] Lustre: Skipped 1 previous similar message [ 7508.105741] Lustre: server umount lustre-MDT0000 complete [ 7512.437601] LustreError: 254724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756024 with bad export cookie 11207660671170691558 [ 7512.449498] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7512.451621] LustreError: 254724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7512.808672] Lustre: server umount lustre-MDT0001 complete [ 7525.267434] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 7528.784671] Key type lgssc unregistered [ 7529.228165] LNet: 260158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7529.239052] LNetError: 260158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7529.274641] LNet: Removed LNI 192.168.202.126@tcp [ 7530.662617] Key type .llcrypt unregistered [ 7530.665768] Key type ._llcrypt unregistered [ 7559.924897] Key type ._llcrypt registered [ 7559.927914] Key type .llcrypt registered [ 7560.176412] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 7562.273620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7562.295713] alg: No test for adler32 (adler32-zlib) [ 7563.702987] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 7564.132451] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 7565.896812] Key type lgssc registered [ 7568.451908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7579.943530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7579.970831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7581.532243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7586.239175] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7590.892271] LustreError: 261584:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7590.910076] LustreError: 261584:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7596.010784] LustreError: 261583:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7596.826045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7597.409798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7602.925112] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7611.893640] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7618.509490] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7626.942821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7627.348875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7634.107698] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7643.086553] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7650.926675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7685.088981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7685.095289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7685.108743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7687.137563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7687.139864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7687.167939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7689.019549] Lustre: server umount lustre-OST0000 complete [ 7694.822274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7694.828170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7694.840296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7694.844751] Lustre: Skipped 1 previous similar message [ 7697.381108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7697.389649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7700.015141] Lustre: server umount lustre-MDT0000 complete [ 7702.497896] LustreError: 262311:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7702.527837] LustreError: 262311:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7705.369077] LustreError: 262318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756217 with bad export cookie 16715265565369180736 [ 7705.390733] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7705.407063] LustreError: 262318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7705.988217] Lustre: server umount lustre-MDT0001 complete [ 7720.495566] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 7725.071783] Key type lgssc unregistered [ 7725.547919] LNet: 265917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7725.565413] LNetError: 265917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7725.581092] LNet: Removed LNI 192.168.202.126@tcp [ 7726.888397] Key type .llcrypt unregistered [ 7726.891959] Key type ._llcrypt unregistered [ 7752.668429] Key type ._llcrypt registered [ 7752.676055] Key type .llcrypt registered [ 7752.827861] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 7754.362888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7754.468212] alg: No test for adler32 (adler32-zlib) [ 7755.666714] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 7755.921740] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 7757.591784] Key type lgssc registered [ 7759.280048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7769.612851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7769.625723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7771.410538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7775.673648] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7780.840418] LustreError: 267137:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7780.852566] LustreError: 267137:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7785.953305] LustreError: 267136:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7786.374328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7787.027372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7793.504920] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7803.785902] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7811.184751] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7818.736373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7819.303988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7826.456534] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7836.024847] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7840.824591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7853.029915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7853.035954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7853.055601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7856.103340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7856.108824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7856.144816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7857.811680] Lustre: server umount lustre-OST0000 complete [ 7863.778555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7863.787302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7863.807795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7863.824269] Lustre: Skipped 1 previous similar message [ 7866.337108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7866.345890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7866.374179] Lustre: Skipped 1 previous similar message [ 7868.826596] Lustre: server umount lustre-MDT0000 complete [ 7871.456339] LustreError: 267133:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7871.467296] LustreError: 267133:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7873.133941] LustreError: 267117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756385 with bad export cookie 16436702601519397696 [ 7873.147224] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7873.158942] LustreError: 267117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7873.766379] Lustre: server umount lustre-MDT0001 complete [ 7885.626567] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 7889.389913] Key type lgssc unregistered [ 7889.648182] LNet: 270816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7889.652914] LNetError: 270816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7889.676732] LNet: Removed LNI 192.168.202.126@tcp [ 7890.561233] Key type .llcrypt unregistered [ 7890.563442] Key type ._llcrypt unregistered [ 7918.750992] Key type ._llcrypt registered [ 7918.752364] Key type .llcrypt registered [ 7918.833638] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 7920.129466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7920.153572] alg: No test for adler32 (adler32-zlib) [ 7921.634452] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 7921.918047] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 7923.607248] Key type lgssc registered [ 7925.415260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7935.550793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7935.586913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7937.166293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7942.813048] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7947.748945] LustreError: 272229:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7947.772122] LustreError: 272229:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7952.644323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7952.877333] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7953.104420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7957.854316] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7966.987366] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7972.665925] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7981.223721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7981.929869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7982.980741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7989.611377] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 7998.543383] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8024.040136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8024.041187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8024.053466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8024.068054] Lustre: Skipped 1 previous similar message [ 8024.089433] LustreError: Skipped 1 previous similar message [ 8029.174934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8029.180024] Lustre: Skipped 2 previous similar messages [ 8029.509811] Lustre: server umount lustre-OST0000 complete [ 8034.277940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8034.305073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8034.316149] Lustre: Skipped 1 previous similar message [ 8039.395850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8039.405322] Lustre: Skipped 1 previous similar message [ 8040.044167] Lustre: server umount lustre-MDT0000 complete [ 8043.791046] LustreError: 272208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756556 with bad export cookie 11220973026797544940 [ 8043.799128] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8043.808594] LustreError: 272208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8044.535279] Lustre: server umount lustre-MDT0001 complete [ 8057.285462] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 8060.609900] Key type lgssc unregistered [ 8061.001304] LNet: 276536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8061.013057] LNetError: 276536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8061.046164] LNet: Removed LNI 192.168.202.126@tcp [ 8062.232189] Key type .llcrypt unregistered [ 8062.234443] Key type ._llcrypt unregistered [ 8082.286898] Key type ._llcrypt registered [ 8082.292654] Key type .llcrypt registered [ 8082.434950] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 8084.099618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8084.122515] alg: No test for adler32 (adler32-zlib) [ 8085.175479] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 8085.543346] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 8087.239748] Key type lgssc registered [ 8088.583311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8097.863668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8097.894144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8099.561738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8105.387888] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8109.030160] LustreError: 277769:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8109.074219] LustreError: 277769:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 8114.159918] LustreError: 277771:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.657199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8115.106589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8120.312107] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8128.265889] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8135.061778] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8143.338161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8143.836475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8149.015858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8151.897769] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8160.512805] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8173.626660] Lustre: server umount lustre-OST0000 complete [ 8180.738212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8180.741178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8180.765180] Lustre: Skipped 1 previous similar message [ 8180.785740] Lustre: Skipped 1 previous similar message [ 8185.826987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8185.832653] Lustre: Skipped 1 previous similar message [ 8186.996669] Lustre: server umount lustre-MDT0000 complete [ 8190.945583] Lustre: 277066:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788756687/real 1788756687] req@000000000f34125f x1875647237485696/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788756703 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8190.956559] LustreError: 277781:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8191.009970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8191.044594] LustreError: 277781:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 8191.525145] LustreError: 277736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756703 with bad export cookie 4301637322221302080 [ 8191.551686] LustreError: 277736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8191.558800] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8192.045598] Lustre: server umount lustre-MDT0001 complete [ 8204.317856] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 8208.287336] Key type lgssc unregistered [ 8208.639227] LNet: 281470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8208.649613] LNetError: 281470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8208.686379] LNet: Removed LNI 192.168.202.126@tcp [ 8209.930167] Key type .llcrypt unregistered [ 8209.936276] Key type ._llcrypt unregistered [ 8225.057801] Key type ._llcrypt registered [ 8225.059720] Key type .llcrypt registered [ 8225.191932] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8245.807557] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 8262.731885] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 8264.513500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8264.578572] alg: No test for adler32 (adler32-zlib) [ 8265.770065] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 8266.195731] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 8268.032658] Key type lgssc registered [ 8269.326705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8309.730326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8309.765959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8311.021663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8311.058187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8311.150663] Lustre: lustre-MDT0000: new disk, initializing [ 8311.224225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8311.245685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8316.618215] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8327.390272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8327.560325] Lustre: 285026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8327.635171] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8327.638171] Lustre: Skipped 1 previous similar message [ 8327.680055] Lustre: lustre-MDT0001: new disk, initializing [ 8327.766618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8327.811487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8327.833103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8333.068703] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8346.043719] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8353.966454] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8362.116803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8362.293680] Lustre: lustre-OST0000: new disk, initializing [ 8362.298722] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8362.305800] Lustre: 286994:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8362.416307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8368.696594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8368.703055] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8368.767615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8368.845241] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8378.258375] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8383.359318] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8383.618944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8388.207836] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8388.414198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8394.218338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8394.266427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8396.752405] Lustre: server umount lustre-OST0000 complete [ 8404.452502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8404.459753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8404.470407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8404.479585] Lustre: Skipped 2 previous similar messages [ 8404.490075] Lustre: Skipped 2 previous similar messages [ 8409.569968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8409.588222] Lustre: Skipped 1 previous similar message [ 8414.689339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8414.696468] Lustre: Skipped 1 previous similar message [ 8415.199175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8415.705715] Lustre: server umount lustre-MDT0000 complete [ 8419.662684] LustreError: 285017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756931 with bad export cookie 5634469669794632244 [ 8419.662684] LustreError: 286993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756931 with bad export cookie 5634469669794632244 [ 8419.663474] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8419.678486] LustreError: 285017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8420.039574] Lustre: server umount lustre-MDT0001 complete [ 8428.724571] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 00:55:39 (1788756939) [ 8436.349751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8436.880128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8441.570808] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8442.342920] LustreError: 289406:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8442.369387] LustreError: 289406:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 8447.461590] LustreError: 289405:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8448.929158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8449.224769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8454.025852] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8462.469797] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8470.502434] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8477.573123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8478.001234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8484.165548] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8491.786178] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8504.803111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8504.807250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8504.827632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8504.851448] Lustre: Skipped 1 previous similar message [ 8504.867789] Lustre: Skipped 1 previous similar message [ 8510.648509] Lustre: server umount lustre-OST0000 complete [ 8515.044273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8515.056654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8515.060797] Lustre: Skipped 2 previous similar messages [ 8520.695829] Lustre: server umount lustre-MDT0000 complete [ 8524.581713] LustreError: 289387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788757036 with bad export cookie 5634469669794633126 [ 8524.586611] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8524.593951] LustreError: 289387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8524.829599] Lustre: server umount lustre-MDT0001 complete [ 8536.602050] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 8540.156613] Key type lgssc unregistered [ 8540.476861] LNet: 292959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8540.483736] LNetError: 292959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8540.500850] LNet: Removed LNI 192.168.202.126@tcp [ 8541.537625] Key type .llcrypt unregistered [ 8541.540238] Key type ._llcrypt unregistered [ 8570.014721] Key type ._llcrypt registered [ 8570.017392] Key type .llcrypt registered [ 8570.119482] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 8586.229097] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 8587.474254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8587.555398] alg: No test for adler32 (adler32-zlib) [ 8588.624901] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 8588.847192] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 8590.497794] Key type lgssc registered [ 8592.037467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8642.075866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8642.120933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8643.250494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8643.286161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8643.381911] Lustre: lustre-MDT0000: new disk, initializing [ 8643.517622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8643.533995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8648.047575] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8660.442477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8660.658813] Lustre: 296786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8660.762537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8660.782175] Lustre: Skipped 1 previous similar message [ 8660.873224] Lustre: lustre-MDT0001: new disk, initializing [ 8661.033935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8661.108592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8661.132418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8666.306380] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8675.758908] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8681.912386] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8689.063409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8689.404279] Lustre: lustre-OST2710: new disk, initializing [ 8689.408493] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8689.412845] Lustre: 298758:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8689.500319] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8694.865576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8694.885466] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8695.103908] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8695.678568] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8705.069819] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8711.775056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8711.919571] Lustre: lustre-OST03e8: new disk, initializing [ 8711.921883] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8711.925979] Lustre: 299822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8711.996286] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8719.313204] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8720.498812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8720.523197] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8720.680693] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8727.066924] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8735.188992] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8735.459689] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8740.170735] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8740.444437] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8745.126725] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8745.353606] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8750.433605] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8750.640363] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8756.721146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8756.728533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8756.740065] Lustre: Skipped 2 previous similar messages [ 8756.748965] Lustre: Skipped 3 previous similar messages [ 8762.679828] Lustre: server umount lustre-MDT0000 complete [ 8766.956109] LustreError: 296793:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8766.989036] LustreError: 296793:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 8 previous similar messages [ 8767.154638] LustreError: 296778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788757279 with bad export cookie 11875351615771734957 [ 8767.181380] LustreError: 296778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8767.196816] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8767.674093] Lustre: server umount lustre-MDT0001 complete [ 8781.414795] Lustre: server umount lustre-OST2710 complete [ 8794.153961] Lustre: server umount lustre-OST03e8 complete [ 8813.335581] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 8824.195715] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 8861.153744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8861.488328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8861.529396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8861.628321] Lustre: lustre-MDT0000: new disk, initializing [ 8861.726388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8861.762236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8866.628276] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8878.799933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8878.991206] Lustre: 305083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8879.062191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8879.067161] Lustre: Skipped 1 previous similar message [ 8879.149448] Lustre: lustre-MDT0001: new disk, initializing [ 8879.259622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8879.322502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8879.343277] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8885.193106] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8896.552739] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8904.031769] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8910.629040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8910.890976] Lustre: lustre-OST0000: new disk, initializing [ 8910.894431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8910.907914] Lustre: 307053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8911.058042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8916.606379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8916.612695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8916.692574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8916.977198] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 8926.982122] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8932.843646] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8933.095680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8938.092814] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8938.322100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8942.577076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8942.587808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8942.594375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8942.600020] Lustre: Skipped 2 previous similar messages [ 8942.624240] LustreError: Skipped 1 previous similar message [ 8946.775516] Lustre: server umount lustre-OST0000 complete [ 8951.266685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8951.275201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8951.285526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8951.290238] Lustre: Skipped 1 previous similar message [ 8956.981770] Lustre: server umount lustre-MDT0000 complete [ 8957.934980] LustreError: 305095:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8957.951837] LustreError: 305095:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 8960.852922] LustreError: 305077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788757473 with bad export cookie 11875351615771736637 [ 8960.859854] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8960.868105] LustreError: 305077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8961.531743] Lustre: server umount lustre-MDT0001 complete [ 8970.414675] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 01:04:41 (1788757481) [ 8972.533940] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8975.134690] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 01:04:45 (1788757485) [ 8979.849139] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 9009.239630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.516525] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9009.538728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9009.737137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9014.164591] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9021.751269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9021.805928] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9021.821261] Lustre: 310914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9027.227647] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9049.406283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9049.784913] Lustre: Denying initial registration attempt from nid 192.168.202.126@tcp, specified as failover [ 9049.801571] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9049.816566] LustreError: 312367:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9049.823260] LustreError: 312367:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9049.831815] LustreError: 312367:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9049.953909] Lustre: server umount lustre-OST0000 complete [ 9049.963940] LustreError: 312367:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9061.860494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9061.867462] Lustre: Skipped 1 previous similar message [ 9061.871666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9061.875262] Lustre: Skipped 3 previous similar messages [ 9066.976952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9066.983915] Lustre: Skipped 1 previous similar message [ 9072.095616] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9072.590390] Lustre: server umount lustre-MDT0000 complete [ 9077.226770] LustreError: 312458:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9077.251150] LustreError: 312458:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9078.058871] LustreError: 310906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788757590 with bad export cookie 11875351615771737519 [ 9078.071102] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9078.089508] LustreError: 310906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9078.474194] Lustre: server umount lustre-MDT0001 complete [ 9095.298774] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 9098.469877] Key type lgssc unregistered [ 9098.780886] LNet: 313491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9098.794704] LNetError: 313491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9098.819145] LNet: Removed LNI 192.168.202.126@tcp [ 9099.701214] Key type .llcrypt unregistered [ 9099.702825] Key type ._llcrypt unregistered [ 9113.651478] Key type ._llcrypt registered [ 9113.658935] Key type .llcrypt registered [ 9113.829366] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 9115.593151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9115.697893] alg: No test for adler32 (adler32-zlib) [ 9116.938827] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 9117.272780] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 9119.089750] Key type lgssc registered [ 9120.950792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9172.501205] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 9183.011926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9183.046205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9184.320465] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9184.345107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9184.528251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9189.234402] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9198.016782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9198.110750] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9198.137905] Lustre: 316253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9198.429672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9203.228924] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9211.023655] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9216.746885] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9237.376763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9237.650418] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9237.911945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9244.301797] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9252.037799] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9256.424823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9256.425963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9256.433514] LustreError: Skipped 1 previous similar message [ 9256.438982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9256.447963] Lustre: Skipped 1 previous similar message [ 9256.465166] Lustre: Skipped 1 previous similar message [ 9260.306229] Lustre: server umount lustre-OST0000 complete [ 9265.121236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9265.130811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9265.145264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9266.656441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9266.658078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9266.667798] Lustre: Skipped 1 previous similar message [ 9270.764834] Lustre: server umount lustre-MDT0000 complete [ 9274.785874] LustreError: 316245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788757787 with bad export cookie 11065770857064934619 [ 9274.799040] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9275.236237] Lustre: server umount lustre-MDT0001 complete [ 9286.672786] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 9290.136140] Key type lgssc unregistered [ 9290.552664] LNet: 319825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9290.562022] LNetError: 319825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9290.578932] LNet: Removed LNI 192.168.202.126@tcp [ 9291.433928] Key type .llcrypt unregistered [ 9291.440747] Key type ._llcrypt unregistered [ 9318.138678] Key type ._llcrypt registered [ 9318.141038] Key type .llcrypt registered [ 9318.280405] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 9319.723796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9319.758619] alg: No test for adler32 (adler32-zlib) [ 9320.928665] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 9321.245072] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 9322.976387] Key type lgssc registered [ 9324.096846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9332.048205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9332.073383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9333.957476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9339.910904] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9344.483588] LustreError: 321237:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9344.530707] LustreError: 321237:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 9349.605776] LustreError: 321236:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9350.961637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9351.288684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9356.024534] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9365.086303] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9372.944339] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9383.481133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9384.075882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9392.651782] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9400.420684] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9415.145364] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9415.150570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9415.153099] Lustre: Skipped 2 previous similar messages [ 9415.174322] Lustre: Skipped 2 previous similar messages [ 9417.463627] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.26@tcp (stopping) [ 9420.257147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9420.265643] Lustre: Skipped 2 previous similar messages [ 9421.114112] Lustre: server umount lustre-MDT0000 complete [ 9422.540960] LustreError: 321231:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9424.977568] LustreError: 321218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788757937 with bad export cookie 13025189100694736006 [ 9424.986675] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9424.993255] LustreError: 321218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9425.376454] LustreError: 321232:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9425.376876] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9425.417735] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9429.410213] LustreError: 321233:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9429.438049] LustreError: 321233:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9431.491439] Lustre: server umount lustre-MDT0001 complete [ 9437.861939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9447.476387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9449.917877] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb4c2c47a07332a46 [ 9449.931469] Lustre: MGC192.168.202.126@tcp: Connection restored to 0@lo (at 0@lo) [ 9450.281668] LustreError: 323069:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9450.378864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9450.446934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9455.767493] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9459.597708] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9466.004911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9466.497179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9471.959481] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9471.985937] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9480.823148] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9482.582794] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9488.486661] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9490.204780] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9496.555829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9496.565308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9496.596741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9496.599335] Lustre: Skipped 2 previous similar messages [ 9497.064126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9502.648649] Lustre: server umount lustre-OST0000 complete [ 9507.296518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9507.300664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9507.313349] Lustre: Skipped 1 previous similar message [ 9507.321502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9507.324015] Lustre: Skipped 2 previous similar messages [ 9513.402700] Lustre: server umount lustre-MDT0000 complete [ 9517.111967] LustreError: 324857:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9517.139057] LustreError: 324857:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [ 9518.053777] LustreError: 321218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758030 with bad export cookie 13025189100694743622 [ 9518.058218] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9518.061637] LustreError: 321218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9518.494983] Lustre: server umount lustre-MDT0001 complete [ 9532.707355] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 9535.892199] Key type lgssc unregistered [ 9536.181072] LNet: 327525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9536.193838] LNetError: 327525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9537.253694] LNet: Removed LNI 192.168.202.126@tcp [ 9538.282332] Key type .llcrypt unregistered [ 9538.283935] Key type ._llcrypt unregistered [ 9564.902058] Key type ._llcrypt registered [ 9564.903517] Key type .llcrypt registered [ 9565.036228] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 9566.641546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9566.720961] alg: No test for adler32 (adler32-zlib) [ 9567.854211] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 9568.087819] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 9569.799653] Key type lgssc registered [ 9570.979936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9578.645789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9578.664895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9580.235787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9584.972964] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9590.754808] LustreError: 328938:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9590.774801] LustreError: 328938:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 9594.243422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9594.746827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9600.972353] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9610.559217] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9617.921462] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9624.546240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9624.554322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9624.573359] Lustre: Skipped 1 previous similar message [ 9624.592911] Lustre: Skipped 1 previous similar message [ 9627.663370] Lustre: server umount lustre-MDT0000 complete [ 9629.668372] LustreError: 328939:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9629.686802] LustreError: 328939:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9632.136639] LustreError: 328921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758144 with bad export cookie 6920767322542881979 [ 9632.148605] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9632.498379] Lustre: server umount lustre-MDT0001 complete [ 9649.193452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9649.734455] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9649.803124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9650.087130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9656.513771] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9664.852993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9664.982038] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9665.034278] Lustre: 331528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9665.384286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9669.790342] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9677.839549] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9685.306966] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9696.251111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9696.550488] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9696.569280] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9696.576226] LustreError: 333411:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 9696.580992] LustreError: 333411:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9696.586162] LustreError: 333411:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9696.683776] Lustre: server umount lustre-OST0000 complete [ 9696.687638] LustreError: 333411:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9707.706688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9708.071299] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9708.388810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9715.885379] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9716.451269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9725.125905] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9732.946239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9733.040506] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9733.118581] Lustre: lustre-OST0001: new disk, initializing [ 9733.123441] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9733.130611] Lustre: 334956:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9733.235165] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9738.894808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9738.912452] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9739.029951] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9739.654697] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9747.704176] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9754.592698] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9754.593102] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9754.597090] LustreError: Skipped 1 previous similar message [ 9754.598258] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9754.626877] Lustre: Skipped 1 previous similar message [ 9759.716650] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9759.724371] Lustre: Skipped 2 previous similar messages [ 9764.841767] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9764.851944] Lustre: Skipped 1 previous similar message [ 9765.343138] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9765.481172] Lustre: server umount lustre-OST0001 complete [ 9769.958992] LustreError: 334883:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9769.979960] LustreError: 334883:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9773.027450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9773.036238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9773.039267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9773.057950] Lustre: Skipped 1 previous similar message [ 9773.078445] Lustre: Skipped 1 previous similar message [ 9775.081273] LustreError: 333877:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9775.103088] LustreError: 333877:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9776.866057] Lustre: server umount lustre-OST0000 complete [ 9783.265234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9783.290792] LustreError: Skipped 1 previous similar message [ 9783.297931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9783.324518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9783.327306] Lustre: Skipped 2 previous similar messages [ 9787.401589] Lustre: server umount lustre-MDT0000 complete [ 9790.435363] LustreError: 331539:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9790.457649] LustreError: 331539:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9791.002511] LustreError: 331520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758303 with bad export cookie 6920767322542882539 [ 9791.007327] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9791.012744] LustreError: 331520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9791.042295] LustreError: 328249:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting [ 9791.686684] Lustre: server umount lustre-MDT0001 complete [ 9809.203983] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [ 9812.869756] Key type lgssc unregistered [ 9813.229979] LNet: 336935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9813.237136] LNetError: 336935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9813.251114] LNet: Removed LNI 192.168.202.126@tcp [ 9813.919704] Key type .llcrypt unregistered [ 9813.922424] Key type ._llcrypt unregistered [ 9891.366877] Key type ._llcrypt registered [ 9891.369627] Key type .llcrypt registered [ 9891.506453] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [ 9911.001903] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [ 9912.191253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9912.290782] alg: No test for adler32 (adler32-zlib) [ 9913.614610] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [ 9913.861563] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [ 9915.535260] Key type lgssc registered [ 9916.873603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9954.569858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9954.629921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9955.922518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9955.944552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9955.999899] Lustre: lustre-MDT0000: new disk, initializing [ 9956.056889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9956.068975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9961.134434] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9972.798733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9972.916356] Lustre: 341877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9972.964735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9972.968863] Lustre: Skipped 1 previous similar message [ 9973.075909] Lustre: lustre-MDT0001: new disk, initializing [ 9973.238283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9973.289445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9973.307787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9978.642260] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [ 9988.559671] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9994.741477] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10004.538949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10005.050714] Lustre: lustre-OST0000: new disk, initializing [10005.065506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10005.088074] Lustre: 343844:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10005.242619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10012.269583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10012.279248] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10012.449269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10014.208477] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10025.793489] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10032.362904] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10032.668331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10038.332817] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10038.614623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10043.363380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10043.364296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10043.372973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10043.376662] Lustre: Skipped 1 previous similar message [10048.488492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10048.496789] Lustre: Skipped 1 previous similar message [10053.604387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10055.135151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10055.686901] Lustre: server umount lustre-OST0000 complete [10063.843760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10063.846684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10063.870332] Lustre: Skipped 2 previous similar messages [10063.887218] Lustre: Skipped 2 previous similar messages [10068.964647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10068.978051] Lustre: Skipped 1 previous similar message [10075.103136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10075.337297] Lustre: server umount lustre-MDT0000 complete [10079.297665] LustreError: 343854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758591 with bad export cookie 203833089753591381 [10079.307130] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10079.308543] LustreError: 343854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10079.703091] Lustre: server umount lustre-MDT0001 complete [10088.873183] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 01:23:20 (1788758600) [10096.686253] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 01:23:27 (1788758607) [10107.441367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10108.082584] LustreError: 346603:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10108.123297] LustreError: 346603:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [10108.206854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10112.858117] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10113.507636] LustreError: 346604:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10118.636016] LustreError: 346603:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10122.263324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10122.721280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10129.215887] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10138.871557] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10146.115656] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10153.894702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10154.262985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10161.262393] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10169.702450] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10187.941362] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10189.687714] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10191.314296] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10193.132410] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10195.254656] Lustre: Failing over lustre-MDT0000 [10195.474164] Lustre: server umount lustre-MDT0000 complete [10196.201973] LustreError: 346600:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10198.498412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10198.522958] Lustre: Skipped 1 previous similar message [10201.304694] LustreError: 346597:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10201.335879] LustreError: 346597:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [10211.523315] LustreError: 347699:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10211.531957] LustreError: 347699:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [10213.791258] Lustre: 340102:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788758710/real 1788758710] req@0000000033cf36a7 x1875649155383040/t0(0) o400->MGC192.168.202.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788758726 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10213.827672] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10215.713727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10224.098779] LustreError: 340099:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000050d82b7d x1875649155388032/t0(0) o250->MGC192.168.202.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10224.421826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10224.501561] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10226.897256] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10228.169609] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10229.759955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10229.816771] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10229.881581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10238.576105] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10240.400867] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10244.493667] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10246.855102] Lustre: Failing over lustre-MDT0000 [10247.297259] Lustre: server umount lustre-MDT0000 complete [10247.422495] LustreError: 346600:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10247.458272] LustreError: 346600:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 11 previous similar messages [10250.211972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10250.215618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10250.221793] Lustre: Skipped 1 previous similar message [10265.807792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10266.053435] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10266.273216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10266.313700] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10268.884490] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10271.083751] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10271.727797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10271.733510] Lustre: Skipped 2 previous similar messages [10271.768493] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10271.811381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10282.890803] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10285.245840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10292.203347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10292.207535] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10292.215407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10292.215415] Lustre: Skipped 2 previous similar messages [10292.237867] Lustre: Skipped 4 previous similar messages [10298.555762] Lustre: server umount lustre-MDT0000 complete [10302.433820] LustreError: 347331:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10302.478294] LustreError: 347331:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 18 previous similar messages [10303.480135] LustreError: 346584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758815 with bad export cookie 203833089753594090 [10303.481285] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10303.489218] LustreError: 346584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10303.951744] Lustre: server umount lustre-MDT0001 complete [10319.273557] Lustre: server umount lustre-OST0000 complete [10336.543678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10337.109486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10342.575084] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10352.512041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10359.136326] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10367.664790] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10375.126754] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10382.560805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10389.752794] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10398.038339] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10407.476849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10417.646412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10417.649705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10417.670302] Lustre: Skipped 1 previous similar message [10417.687888] Lustre: Skipped 6 previous similar messages [10422.114331] Lustre: server umount lustre-OST0000 complete [10432.486091] Lustre: server umount lustre-MDT0000 complete [10432.996087] LustreError: 352966:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10433.020052] LustreError: 352966:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [10437.011219] LustreError: 352952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758949 with bad export cookie 203833089753594839 [10437.023077] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10437.029350] LustreError: 352952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10437.494891] Lustre: server umount lustre-MDT0001 complete [10451.156650] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [10454.134690] Key type lgssc unregistered [10454.419237] LNet: 356527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10454.431142] LNetError: 356527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10455.465412] LNet: Removed LNI 192.168.202.126@tcp [10456.721588] Key type .llcrypt unregistered [10456.723936] Key type ._llcrypt unregistered [10485.194804] Key type ._llcrypt registered [10485.203415] Key type .llcrypt registered [10485.401684] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [10486.762498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10486.805573] alg: No test for adler32 (adler32-zlib) [10488.571633] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [10488.990533] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [10490.983485] Key type lgssc registered [10493.274274] Lustre: Echo OBD driver; http://www.lustre.org/ [10502.644872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10502.674731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10504.141786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10509.165804] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10514.401809] LustreError: 358039:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10514.410905] LustreError: 358039:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [10518.762296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10519.088198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10523.909889] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10532.933576] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10538.706656] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10546.311794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10546.808348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10548.882675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10554.519846] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10562.633235] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10578.806444] Lustre: server umount lustre-MDT0000 complete [10579.965297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10579.989075] Lustre: Skipped 2 previous similar messages [10580.005169] LustreError: 358035:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10580.042765] LustreError: 358035:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [10582.785505] LustreError: 358021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759095 with bad export cookie 4682084256948710704 [10582.791824] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10582.798350] LustreError: 358021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10583.226140] Lustre: server umount lustre-MDT0001 complete [10597.254989] Lustre: server umount lustre-OST0000 complete [10611.543767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10612.320188] LustreError: 361773:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10612.457610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10617.829895] LustreError: 361774:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10618.084410] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10621.927868] LustreError: 361773:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10627.601173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10627.988724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10632.904249] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10642.003487] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10647.635731] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10654.833050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10655.227289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10657.282536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10661.550794] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10668.771993] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10683.884767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10683.896351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10683.911323] Lustre: Skipped 1 previous similar message [10683.932147] Lustre: Skipped 2 previous similar messages [10686.778864] Lustre: server umount lustre-MDT0000 complete [10688.998246] LustreError: 364215:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10689.041165] LustreError: 364215:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [10690.951393] LustreError: 361755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759203 with bad export cookie 4682084256948711817 [10690.961419] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10690.968297] LustreError: 361755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10691.498197] Lustre: server umount lustre-MDT0001 complete [10707.170438] Lustre: server umount lustre-OST0000 complete [10724.920240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10725.340649] LustreError: 365687:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10725.415350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10729.810374] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10739.261355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10745.160060] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10753.406597] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10759.772965] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10767.358549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10767.776668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10767.784935] Lustre: Skipped 1 previous similar message [10769.854420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10775.020536] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [10783.687512] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10790.378118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10790.390620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10790.408671] Lustre: Skipped 1 previous similar message [10790.431868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10794.465306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10794.478777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10794.497564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10796.049726] Lustre: server umount lustre-OST0000 complete [10801.120408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10801.128722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10801.146504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10801.149041] Lustre: Skipped 1 previous similar message [10807.427732] Lustre: server umount lustre-MDT0000 complete [10809.829352] LustreError: 365706:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10809.869590] LustreError: 365706:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [10812.266559] LustreError: 365667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759324 with bad export cookie 4682084256948712839 [10812.282851] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10812.292812] LustreError: 365667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10812.961730] Lustre: server umount lustre-MDT0001 complete [10825.893151] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [10829.670593] Key type lgssc unregistered [10830.198641] LNet: 369221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10830.219842] LNetError: 369221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10831.273336] LNet: Removed LNI 192.168.202.126@tcp [10832.577730] Key type .llcrypt unregistered [10832.580142] Key type ._llcrypt unregistered [10865.409249] Key type ._llcrypt registered [10865.418959] Key type .llcrypt registered [10865.556275] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [10867.092667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10867.411105] alg: No test for adler32 (adler32-zlib) [10868.726989] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [10869.061642] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [10870.961784] Key type lgssc registered [10872.543636] Lustre: Echo OBD driver; http://www.lustre.org/ [10882.838738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10882.880996] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10882.892948] LustreError: 370661:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10882.916064] LustreError: 370661:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10882.920077] LustreError: 370661:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10882.944447] LustreError: 370661:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10882.957369] LustreError: 370661:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10891.089295] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10891.096260] LustreError: 371058:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10891.111206] LustreError: 371058:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10891.119522] LustreError: 371058:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10891.131709] LustreError: 371058:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10891.143839] LustreError: 371058:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10912.796731] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [10916.586879] Key type lgssc unregistered [10916.992223] LNet: 372062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10916.999292] LNetError: 372062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10917.018587] LNet: Removed LNI 192.168.202.126@tcp [10917.974535] Key type .llcrypt unregistered [10917.976458] Key type ._llcrypt unregistered [10938.637905] Key type ._llcrypt registered [10938.639784] Key type .llcrypt registered [10938.746208] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [10956.030311] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [10957.526374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10957.578864] alg: No test for adler32 (adler32-zlib) [10959.104472] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [10959.517734] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [10961.273124] Key type lgssc registered [10962.855958] Lustre: Echo OBD driver; http://www.lustre.org/ [11002.830467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11002.872042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11004.140433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11004.190616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11004.288732] Lustre: lustre-MDT0000: new disk, initializing [11004.373403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11004.388630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11009.196760] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11020.748680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11020.942216] Lustre: 375214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11020.995839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11020.999237] Lustre: Skipped 1 previous similar message [11021.059262] Lustre: lustre-MDT0001: new disk, initializing [11021.160655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11021.206078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11021.226697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11027.137793] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11038.126167] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11044.808916] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11052.185048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11052.444901] Lustre: lustre-OST0000: new disk, initializing [11052.449265] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11052.459871] Lustre: 377185:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11052.547706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11058.952935] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11059.342068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11059.362700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11059.480027] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11068.021801] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11072.942165] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11073.173962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11077.093128] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11077.234366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11080.161155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11080.161740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11080.174181] LustreError: Skipped 1 previous similar message [11080.177169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11080.205367] Lustre: Skipped 1 previous similar message [11085.070263] Lustre: server umount lustre-OST0000 complete [11090.400530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11090.437339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11090.440034] Lustre: Skipped 1 previous similar message [11095.250700] Lustre: server umount lustre-MDT0000 complete [11099.751047] LustreError: 375206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759611 with bad export cookie 14533263184281189440 [11099.763057] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11099.766596] LustreError: 375206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11100.413782] Lustre: server umount lustre-MDT0001 complete [11110.046184] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11112.334177] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 01:40:23 (1788759623) [11120.099651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11120.638459] LustreError: 379642:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11120.649959] LustreError: 379642:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [11120.702594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11125.692675] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11125.730357] LustreError: 379643:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11129.828433] LustreError: 379642:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11134.957171] LustreError: 379643:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11135.516087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11136.118525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11141.405639] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11150.287686] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11156.893695] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11164.387292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11164.704390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11171.360455] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11178.789798] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11186.100055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11186.396812] Lustre: lustre-OST0001: new disk, initializing [11186.402414] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11186.413040] Lustre: 382480:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11186.515618] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11194.089431] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11196.027491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11196.044819] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11196.172957] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11206.035829] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11211.763608] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11211.770235] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11211.780624] LustreError: Skipped 1 previous similar message [11211.781727] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11211.823556] Lustre: Skipped 2 previous similar messages [11216.261859] Lustre: server umount lustre-OST0001 complete [11216.895966] LustreError: 381466:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11216.921761] LustreError: 381466:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [11227.105136] LustreError: 381466:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11227.119872] LustreError: 381466:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [11237.358798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11237.366988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11237.380203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11237.396708] Lustre: Skipped 3 previous similar messages [11240.511918] Lustre: server umount lustre-OST0000 complete [11247.584952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11247.601170] Lustre: Skipped 1 previous similar message [11247.609536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11247.618161] Lustre: Skipped 1 previous similar message [11251.074034] Lustre: server umount lustre-MDT0000 complete [11252.706777] LustreError: 379642:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11252.718165] LustreError: 379642:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [11255.252781] LustreError: 379624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759767 with bad export cookie 14533263184281190322 [11255.257170] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11255.261490] LustreError: 379624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11261.808480] Lustre: server umount lustre-MDT0001 complete [11274.585516] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [11278.077427] Key type lgssc unregistered [11278.581523] LNet: 384467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11278.594607] LNetError: 384467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11278.615403] LNet: Removed LNI 192.168.202.126@tcp [11279.608386] Key type .llcrypt unregistered [11279.614351] Key type ._llcrypt unregistered [11335.873763] Key type ._llcrypt registered [11335.880175] Key type .llcrypt registered [11336.071246] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [11337.259858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11337.297790] alg: No test for adler32 (adler32-zlib) [11338.985960] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [11339.315204] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [11341.082109] Key type lgssc registered [11342.523599] Lustre: Echo OBD driver; http://www.lustre.org/ [11352.519803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11352.555099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11353.769798] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11353.787466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11353.941707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11358.827373] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11367.459241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11367.553553] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11367.574214] Lustre: 387261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11367.680746] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11367.914797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11373.129566] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11381.853780] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11387.999365] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11393.504194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11393.506044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11393.531993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11393.537120] Lustre: Skipped 1 previous similar message [11393.564883] Lustre: Skipped 1 previous similar message [11396.677652] Lustre: server umount lustre-MDT0000 complete [11400.545171] LustreError: 387254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759912 with bad export cookie 12371314195275375328 [11400.563187] LustreError: 387254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11400.563949] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11401.253577] Lustre: server umount lustre-MDT0001 complete [11412.865455] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 01:45:23 (1788759923) [11420.118886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11420.476661] LustreError: 389775:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11420.489720] LustreError: 389775:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [11420.577254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11425.567316] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11425.770844] LustreError: 389776:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11430.879874] LustreError: 389775:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11435.733736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11435.978955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11442.141901] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11451.271877] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11459.502209] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11467.272432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11467.544703] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11467.779241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11474.933696] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11483.476715] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11490.953388] Lustre: Permanently deactivating lustre-OST0000 [11490.971962] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11499.581282] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11500.883878] LustreError: 392762:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11500.891293] LustreError: 392762:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11500.896300] LustreError: 392762:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11504.286487] LustreError: 392811:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11504.290534] LustreError: 392811:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11512.444690] Lustre: server umount lustre-OST0000 complete [11515.838270] LustreError: 393060:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11515.842993] LustreError: 393060:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11517.920435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11517.936886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11517.953833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11519.458607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11524.586750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11524.609973] Lustre: Skipped 3 previous similar messages [11529.697465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11529.711401] Lustre: Skipped 1 previous similar message [11532.255121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11532.656402] Lustre: server umount lustre-MDT0000 complete [11534.821937] LustreError: 392393:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11534.840221] LustreError: 392393:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [11536.762768] LustreError: 389757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760048 with bad export cookie 12371314195275375909 [11536.766239] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11536.779955] LustreError: 389757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11547.290542] Lustre: server umount lustre-MDT0001 complete [11556.819504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11562.522407] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11565.808518] LustreError: 394221:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11572.775367] Lustre: 394560:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.126@tcp [11575.258165] Lustre: server umount MGS complete [11587.414639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11587.977200] LustreError: 395283:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11588.002604] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11588.015180] Lustre: Skipped 1 previous similar message [11588.117559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11593.626372] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11598.310734] LustreError: 395283:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11598.336219] LustreError: 395283:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [11602.733910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11603.147899] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11608.741293] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11617.136292] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11623.882025] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11626.125414] Lustre: Permanently reactivating lustre-OST0000 [11626.137590] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11626.141832] Lustre: Skipped 2 previous similar messages [11652.580056] LustreError: 397014:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11661.671287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11662.178763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11662.185105] Lustre: Skipped 1 previous similar message [11669.969419] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11678.322397] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11686.033276] Lustre: server umount lustre-OST0000 complete [11690.467361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11690.486228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11690.500204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11700.316895] Lustre: server umount lustre-MDT0000 complete [11703.268926] LustreError: 395284:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11703.302366] LustreError: 395284:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [11704.290669] Lustre: 386585:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788760200/real 1788760200] req@00000000d5bff137 x1875650649625472/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788760216 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11704.332345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11704.363980] Lustre: Skipped 1 previous similar message [11704.585241] LustreError: 395264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760216 with bad export cookie 12371314195275376959 [11704.589586] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11704.598047] LustreError: 395264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11704.955416] Lustre: server umount lustre-MDT0001 complete [11716.491713] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [11719.374721] Key type lgssc unregistered [11719.790285] LNet: 399599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11719.794943] LNetError: 399599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11719.818903] LNet: Removed LNI 192.168.202.126@tcp [11720.691259] Key type .llcrypt unregistered [11720.693522] Key type ._llcrypt unregistered [11741.527619] Key type ._llcrypt registered [11741.530841] Key type .llcrypt registered [11741.633858] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [11761.054853] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [11762.220165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11762.421397] alg: No test for adler32 (adler32-zlib) [11763.516932] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [11763.980412] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [11765.767449] Key type lgssc registered [11767.448037] Lustre: Echo OBD driver; http://www.lustre.org/ [11806.650898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11806.703233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11807.912994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11807.927388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11807.971829] Lustre: lustre-MDT0000: new disk, initializing [11808.044624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11808.057344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11812.648962] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11824.737457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11824.938303] Lustre: 402756:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11825.052494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11825.055494] Lustre: Skipped 1 previous similar message [11825.199381] Lustre: lustre-MDT0001: new disk, initializing [11825.351326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11825.397706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11825.420310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11831.414777] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11841.786581] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11848.291223] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11855.425340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11855.675574] Lustre: lustre-OST0000: new disk, initializing [11855.678339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11855.683550] Lustre: 404726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11855.759628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11863.031576] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11864.698917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11864.713959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11864.884018] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11874.813729] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11880.393823] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11880.594087] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11885.825637] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11886.086392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11890.674943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11890.678388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11890.692070] LustreError: Skipped 1 previous similar message [11890.709859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11894.350145] Lustre: server umount lustre-OST0000 complete [11900.899223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11900.919152] Lustre: Skipped 2 previous similar messages [11900.919717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11900.940928] Lustre: Skipped 2 previous similar messages [11904.818261] Lustre: server umount lustre-MDT0000 complete [11909.497684] LustreError: 402749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760421 with bad export cookie 14535117812431095236 [11909.514217] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11909.987432] Lustre: server umount lustre-MDT0001 complete [11919.663734] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11921.704164] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 01:53:52 (1788760432) [11929.839658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11930.373031] LustreError: 407187:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11930.391431] LustreError: 407187:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [11930.454661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11935.712931] LustreError: 407188:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11936.089903] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11939.813970] LustreError: 407187:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11944.929294] LustreError: 407188:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11945.276853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11945.661742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11950.057960] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11957.484714] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11964.643793] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11972.934471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11973.255308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11979.687817] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [11986.755911] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11988.612343] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11992.339833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12001.252880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12001.260033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12001.262980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12001.274368] Lustre: Skipped 1 previous similar message [12002.997983] Lustre: server umount lustre-OST0000 complete [12011.489718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12011.494136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12011.512737] Lustre: Skipped 1 previous similar message [12011.523916] Lustre: Skipped 2 previous similar messages [12013.466604] Lustre: server umount lustre-MDT0000 complete [12016.609951] LustreError: 407184:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12016.635858] LustreError: 407184:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12017.148781] LustreError: 407167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760529 with bad export cookie 14535117812431096118 [12017.155104] LustreError: 407167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12017.162095] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12017.518384] Lustre: server umount lustre-MDT0001 complete [12030.355828] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [12033.715873] Key type lgssc unregistered [12034.024866] LNet: 410910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12034.035438] LNetError: 410910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12035.116904] LNet: Removed LNI 192.168.202.126@tcp [12036.191844] Key type .llcrypt unregistered [12036.194238] Key type ._llcrypt unregistered [12061.899253] Key type ._llcrypt registered [12061.905263] Key type .llcrypt registered [12062.081313] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [12062.371435] Key type .llcrypt unregistered [12062.373120] Key type ._llcrypt unregistered [12082.188783] Key type ._llcrypt registered [12082.194081] Key type .llcrypt registered [12082.410959] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [12083.816936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12083.969780] alg: No test for adler32 (adler32-zlib) [12085.142953] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [12085.441261] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [12087.186745] Key type lgssc registered [12088.504537] Lustre: Echo OBD driver; http://www.lustre.org/ [12097.415346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12097.431839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12098.832615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12103.424659] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12109.290822] LustreError: 413118:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12109.301436] LustreError: 413118:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [12112.659210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12112.927241] LustreError: 413113:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12112.940027] LustreError: 413113:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12112.954808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12117.986391] LustreError: 413118:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12118.004359] LustreError: 413118:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12120.209662] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12123.110070] LustreError: 413119:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12123.143246] LustreError: 413119:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12124.189699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12128.659708] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12129.248716] LustreError: 413118:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12129.284160] LustreError: 413118:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [12135.421257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12135.819420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12140.392267] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12149.730900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12149.754277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12149.778664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12150.786707] Lustre: server umount lustre-OST0000 complete [12155.427906] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12155.431208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12155.453620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12155.786928] Lustre: server umount lustre-MDT0000 complete [12156.392993] LustreError: 414856:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12156.415100] LustreError: 414856:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [12159.697231] LustreError: 413099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760671 with bad export cookie 8831002691926113410 [12159.700752] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12159.704408] LustreError: 413099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12169.354286] Lustre: server umount lustre-MDT0001 complete [12184.130477] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [12187.561459] Key type lgssc unregistered [12187.885972] LNet: 416265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12187.893489] LNetError: 416265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12187.907589] LNet: Removed LNI 192.168.202.126@tcp [12189.071447] Key type .llcrypt unregistered [12189.073818] Key type ._llcrypt unregistered [12221.715460] Key type ._llcrypt registered [12221.727558] Key type .llcrypt registered [12221.899538] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [12224.090064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12224.136396] alg: No test for adler32 (adler32-zlib) [12225.334368] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [12225.553458] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [12227.279323] Key type lgssc registered [12228.268695] Lustre: Echo OBD driver; http://www.lustre.org/ [12236.546514] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12236.559556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12252.063577] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12257.184306] LustreError: 417656:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.126@tcp: failed processing log, type 4: rc = -110 [12283.871211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12291.481886] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12299.576328] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12307.091781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12307.704162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12307.824349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12312.348698] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12313.059490] LustreError: 417682:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12313.085043] LustreError: 417682:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [12318.185550] LustreError: 418691:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12318.198195] LustreError: 418691:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12321.535294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12321.938529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12327.186530] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12335.217906] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12341.319211] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12348.897635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12348.911648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12348.925191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12350.560862] Lustre: server umount lustre-OST0000 complete [12356.082722] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12356.088225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12356.104607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12356.460149] Lustre: server umount lustre-MDT0000 complete [12358.116237] LustreError: 420486:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12358.125881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12360.374898] LustreError: 417663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760872 with bad export cookie 9950180130324687048 [12360.376901] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12360.383586] LustreError: 417663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12370.144567] Lustre: server umount lustre-MDT0001 complete [12380.309372] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [12383.784879] Key type lgssc unregistered [12384.013314] LNet: 421236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12384.039920] LNetError: 421236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12384.070779] LNet: Removed LNI 192.168.202.126@tcp [12385.217980] Key type .llcrypt unregistered [12385.220525] Key type ._llcrypt unregistered [12416.657155] Key type ._llcrypt registered [12416.664967] Key type .llcrypt registered [12416.877735] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [12418.893558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12418.996153] alg: No test for adler32 (adler32-zlib) [12420.202148] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [12420.568072] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [12422.399196] Key type lgssc registered [12425.295825] Lustre: Echo OBD driver; http://www.lustre.org/ [12434.770498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12434.794683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12436.292791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12441.712287] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12446.697044] LustreError: 422651:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12446.711984] LustreError: 422651:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [12451.659229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12451.813854] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12451.994844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12457.775270] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12466.669682] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12472.970471] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12480.105394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12480.383965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12481.469038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12486.173256] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12494.613728] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12501.986271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12501.991691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12502.019395] Lustre: Skipped 2 previous similar messages [12502.043068] Lustre: Skipped 2 previous similar messages [12502.792083] LustreError: 422647:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12502.876633] Lustre: server umount lustre-MDT0000 complete [12506.578502] LustreError: 422631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.26@tcp arrived at 1788761018 with bad export cookie 17419291014767796731 [12507.122776] LustreError: 422651:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12507.132254] LustreError: 422651:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [12509.811356] LustreError: 422633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788761022 with bad export cookie 17419291014767795842 [12509.836541] LustreError: 422633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12509.853144] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12512.224087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12512.233733] LustreError: 422652:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12512.261357] LustreError: 422652:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [12512.498959] Lustre: server umount lustre-OST0000 complete [12519.631694] Lustre: server umount lustre-MDT0001 complete [12532.589627] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [12536.252972] Key type lgssc unregistered [12536.588391] LNet: 426336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12536.603572] LNetError: 426336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12536.624050] LNet: Removed LNI 192.168.202.126@tcp [12537.624603] Key type .llcrypt unregistered [12537.629396] Key type ._llcrypt unregistered [12565.297263] Key type ._llcrypt registered [12565.299245] Key type .llcrypt registered [12565.404217] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [12566.667631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12566.724653] alg: No test for adler32 (adler32-zlib) [12567.792867] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [12567.946176] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [12569.631944] Key type lgssc registered [12571.125351] Lustre: Echo OBD driver; http://www.lustre.org/ [12580.730896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12580.755258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12582.741800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12588.807809] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12592.099465] LustreError: 427754:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12592.125751] LustreError: 427754:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [12597.225457] LustreError: 427755:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12599.800926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12600.107660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12607.532764] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12616.011889] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12621.981331] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12629.949318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12630.448632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12637.987197] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12646.351824] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12651.959301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12656.618516] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12656.623521] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12656.638169] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12657.129088] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12657.148602] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12657.152522] Lustre: Skipped 1 previous similar message [12661.501991] Lustre: server umount lustre-MDT0001 complete [12662.247681] LustreError: 427755:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12662.269729] LustreError: 427755:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12667.365055] LustreError: 428483:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12667.393669] LustreError: 428483:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [12672.481714] LustreError: 427754:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12672.492355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12672.494329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12672.516138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12672.536130] Lustre: Skipped 1 previous similar message [12675.448245] Lustre: server umount lustre-OST0000 complete [12680.154481] Lustre: server umount lustre-MDT0000 complete [12696.556458] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [12699.553533] Key type lgssc unregistered [12699.864071] LNet: 431440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12699.874164] LNetError: 431440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12699.896026] LNet: Removed LNI 192.168.202.126@tcp [12700.861570] Key type .llcrypt unregistered [12700.864157] Key type ._llcrypt unregistered [12727.965252] Key type ._llcrypt registered [12727.968191] Key type .llcrypt registered [12728.136179] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [12744.930587] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [12746.143028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12746.169544] alg: No test for adler32 (adler32-zlib) [12747.258867] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [12747.428515] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [12749.095822] Key type lgssc registered [12750.750133] Lustre: Echo OBD driver; http://www.lustre.org/ [12795.711410] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [12813.283883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12813.315967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12814.689528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12814.734123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12814.898884] Lustre: lustre-MDT0000: new disk, initializing [12815.002929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12815.042656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12820.204300] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12838.182368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12838.427352] Lustre: lustre-OST0000: new disk, initializing [12838.437896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12838.442276] Lustre: Skipped 1 previous similar message [12838.451407] Lustre: 436469:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12838.559105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12844.662142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12844.670402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12844.714099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12845.745457] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12856.219315] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12873.310708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12873.454363] Lustre: 435451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12873.514706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12873.567558] Lustre: lustre-MDT0001: new disk, initializing [12873.671968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12873.717460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12873.738508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12879.330631] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [12890.116411] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12890.419822] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12896.038901] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12896.318835] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12901.856744] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12901.863883] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12901.880860] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12901.885958] Lustre: Skipped 1 previous similar message [12904.176937] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.26@tcp (stopping) [12906.978565] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12906.999991] Lustre: Skipped 1 previous similar message [12909.272660] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.26@tcp (stopping) [12914.405134] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.26@tcp (stopping) [12914.427216] Lustre: Skipped 2 previous similar messages [12916.191202] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12916.441600] Lustre: server umount lustre-MDT0001 complete [12922.362511] LustreError: 435458:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12922.405840] LustreError: 435458:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [12927.462877] LustreError: 436765:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12927.496770] LustreError: 436765:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12932.599957] LustreError: 435458:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12932.632147] LustreError: 435458:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12936.693861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12936.703663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12936.721588] Lustre: Skipped 1 previous similar message [12936.727073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12937.711249] LustreError: 435463:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12940.880185] Lustre: server umount lustre-OST0000 complete [12942.826241] LustreError: 435464:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12951.564403] Lustre: server umount lustre-MDT0000 complete [12968.460649] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [12971.882743] Key type lgssc unregistered [12972.365656] LNet: 440218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12972.375888] LNetError: 440218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12973.417209] LNet: Removed LNI 192.168.202.126@tcp [12974.507403] Key type .llcrypt unregistered [12974.509787] Key type ._llcrypt unregistered [13024.926561] Key type ._llcrypt registered [13024.933028] Key type .llcrypt registered [13025.119726] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [13025.604513] Key type .llcrypt unregistered [13025.606685] Key type ._llcrypt unregistered [13045.849209] Key type ._llcrypt registered [13045.851432] Key type .llcrypt registered [13046.034643] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [13047.446621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13047.581086] alg: No test for adler32 (adler32-zlib) [13049.169139] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [13049.522473] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [13051.215235] Key type lgssc registered [13052.541781] Lustre: Echo OBD driver; http://www.lustre.org/ [13084.468482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13084.497042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13085.819711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13085.855163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13085.993157] Lustre: lustre-MDT0000: new disk, initializing [13086.169145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13086.204272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13091.820163] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13103.847994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13104.064145] Lustre: 443590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13104.202931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13104.208622] Lustre: Skipped 1 previous similar message [13104.299042] Lustre: lustre-MDT0001: new disk, initializing [13104.415648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13104.484397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13104.498448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13109.300719] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13120.237343] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13127.034460] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13135.646536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13136.005390] Lustre: lustre-OST0000: new disk, initializing [13136.025338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13136.032610] Lustre: 445557:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13136.139296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13141.542874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13141.551734] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13141.667552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13143.520661] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13154.952562] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13166.051126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13166.067151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13166.082285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13167.590293] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13167.594162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13167.613632] Lustre: Skipped 1 previous similar message [13167.630509] Lustre: Skipped 2 previous similar messages [13169.602423] Lustre: server umount lustre-MDT0000 complete [13174.922607] LustreError: 443582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788761687 with bad export cookie 193476310161207701 [13174.932830] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13174.944989] LustreError: 443582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13175.386597] Lustre: server umount lustre-MDT0001 complete [13190.146629] Lustre: server umount lustre-OST0000 complete [13204.494069] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 02:15:15 (1788761715) [13213.574937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13214.057539] LustreError: 447590:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13214.073160] LustreError: 447590:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [13214.144192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13219.306578] LustreError: 447591:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13219.736376] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13224.418614] LustreError: 447590:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13229.540364] LustreError: 447591:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13230.728479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13231.312409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13237.264705] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13246.434571] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13252.755632] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13274.237245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13274.518738] Lustre: Found index 0 for lustre-OST0000, updating log [13274.697336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13282.262556] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13286.018087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13291.000520] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13296.612739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13296.614798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13296.614902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13296.624200] Lustre: Skipped 1 previous similar message [13296.645665] Lustre: Skipped 1 previous similar message [13302.702439] Lustre: server umount lustre-OST0000 complete [13319.138982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13319.144753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13319.148545] Lustre: Skipped 1 previous similar message [13319.161958] Lustre: Skipped 3 previous similar messages [13324.264457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13324.272389] Lustre: Skipped 1 previous similar message [13325.651192] Lustre: server umount lustre-MDT0000 complete [13330.115756] LustreError: 447570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788761842 with bad export cookie 193476310161209094 [13330.126197] LustreError: 447570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13330.126446] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13330.599303] Lustre: server umount lustre-MDT0001 complete [13337.327694] LNet: 1 local NIs in recovery (showing 1): 192.168.202.126@tcp [13342.247161] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 02:17:32 (1788761852) [13968.689616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13969.180954] LustreError: 454007:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13969.259380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13973.569879] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13974.506416] LustreError: 454008:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13979.624491] LustreError: 454007:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13982.073477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13982.506348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13987.508174] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [13998.095956] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14004.454291] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14024.365068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14024.706578] Lustre: Found index 0 for lustre-OST0000, updating log [14025.040553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14029.604447] LNet: 1 local NIs in recovery (showing 1): 192.168.202.126@tcp [14031.301645] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [14038.462973] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14040.545163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14040.560123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14040.573832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14040.580795] Lustre: Skipped 2 previous similar messages [14054.367264] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14054.635100] Lustre: server umount lustre-OST0000 complete [14059.299237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14059.308556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14059.316335] Lustre: Skipped 1 previous similar message [14061.345102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14061.350368] Lustre: Skipped 2 previous similar messages [14064.496932] LustreError: 454004:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14064.526206] LustreError: 454004:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [14064.678653] Lustre: server umount lustre-MDT0000 complete [14068.397306] LustreError: 453987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788762580 with bad export cookie 193476310161210172 [14068.411416] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14068.417933] LustreError: 453987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14068.879516] Lustre: server umount lustre-MDT0001 complete [14089.055360] LNet: 1 local NIs in recovery (showing 1): 192.168.202.126@tcp [14099.842874] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [14110.997880] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [14150.340899] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 02:31:00 (1788762660) [14172.441943] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [14175.137315] Key type lgssc unregistered [14175.480820] LNet: 460620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14175.486767] LNetError: 460620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14288.174234] LNet: Removed LNI 192.168.202.126@tcp [14289.378507] Key type .llcrypt unregistered [14289.380369] Key type ._llcrypt unregistered [14308.128352] Key type ._llcrypt registered [14308.130599] Key type .llcrypt registered [14308.258722] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [14309.670512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14309.978421] alg: No test for adler32 (adler32-zlib) [14311.149642] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [14311.536148] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [14313.255360] Key type lgssc registered [14314.460267] Lustre: Echo OBD driver; http://www.lustre.org/ [14319.612867] LNet: Added LNI 192.168.202.126@tcp100 [8/256/0/180] [14320.635156] LNet: Added LNI 192.168.202.126@tcp101 [8/256/0/180] [14321.625446] LNet: Added LNI 192.168.202.126@tcp102 [8/256/0/180] [14322.710289] LNet: Added LNI 192.168.202.126@tcp103 [8/256/0/180] [14323.658567] LNet: Added LNI 192.168.202.126@tcp104 [8/256/0/180] [14324.515664] LNet: Added LNI 192.168.202.126@tcp105 [8/256/0/180] [14325.544159] LNet: Added LNI 192.168.202.126@tcp106 [8/256/0/180] [14326.428293] LNet: Added LNI 192.168.202.126@tcp107 [8/256/0/180] [14327.352769] LNet: Added LNI 192.168.202.126@tcp108 [8/256/0/180] [14328.306742] LNet: Added LNI 192.168.202.126@tcp109 [8/256/0/180] [14329.342162] LNet: Added LNI 192.168.202.126@tcp110 [8/256/0/180] [14330.366979] LNet: Added LNI 192.168.202.126@tcp111 [8/256/0/180] [14331.312536] LNet: Added LNI 192.168.202.126@tcp112 [8/256/0/180] [14332.144387] LNet: Added LNI 192.168.202.126@tcp113 [8/256/0/180] [14333.146940] LNet: Added LNI 192.168.202.126@tcp114 [8/256/0/180] [14334.114595] LNet: Added LNI 192.168.202.126@tcp115 [8/256/0/180] [14335.304293] LNet: Added LNI 192.168.202.126@tcp116 [8/256/0/180] [14336.343803] LNet: Added LNI 192.168.202.126@tcp117 [8/256/0/180] [14337.303193] LNet: Added LNI 192.168.202.126@tcp118 [8/256/0/180] [14338.045536] LNet: Added LNI 192.168.202.126@tcp119 [8/256/0/180] [14338.819602] LNet: Added LNI 192.168.202.126@tcp120 [8/256/0/180] [14339.702641] LNet: Added LNI 192.168.202.126@tcp121 [8/256/0/180] [14340.592355] LNet: Added LNI 192.168.202.126@tcp122 [8/256/0/180] [14341.503527] LNet: Added LNI 192.168.202.126@tcp123 [8/256/0/180] [14342.326619] LNet: Added LNI 192.168.202.126@tcp124 [8/256/0/180] [14343.123232] LNet: Added LNI 192.168.202.126@tcp125 [8/256/0/180] [14344.002654] LNet: Added LNI 192.168.202.126@tcp126 [8/256/0/180] [14344.817179] LNet: Added LNI 192.168.202.126@tcp127 [8/256/0/180] [14345.712258] LNet: Added LNI 192.168.202.126@tcp128 [8/256/0/180] [14346.643908] LNet: Added LNI 192.168.202.126@tcp129 [8/256/0/180] [14347.609180] LNet: Added LNI 192.168.202.126@tcp130 [8/256/0/180] [14348.528194] LNet: Added LNI 192.168.202.126@tcp131 [8/256/0/180] [14349.403686] LNet: Added LNI 192.168.202.126@tcp132 [8/256/0/180] [14350.445544] LNet: Added LNI 192.168.202.126@tcp133 [8/256/0/180] [14351.350153] LNet: Added LNI 192.168.202.126@tcp134 [8/256/0/180] [14352.272909] LNet: Added LNI 192.168.202.126@tcp135 [8/256/0/180] [14358.646991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14358.698849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14359.864097] Lustre: 463648:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.126@tcp: can't add NID 192.168.202.126@tcp130: rc = -75 [14359.955252] LNetError: 461103:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.126 [14359.974827] LNetError: Protocol error connecting to 192.168.202.126@tcp118 at host 192.168.202.126:988: is it running a compatible version of Lustre? [14359.997805] LNetError: Skipped 3 previous similar messages [14360.117485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14360.175293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14360.275738] Lustre: lustre-MDT0000: new disk, initializing [14360.415244] Lustre: 463703:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.126@tcp: can't add NID 192.168.202.126@tcp130: rc = -75 [14360.434128] Lustre: 463703:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14360.468626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14360.496967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14360.674793] LNet: 3 local NIs in recovery (showing 3): 192.168.202.126@tcp128, 192.168.202.126@tcp129, 192.168.202.126@tcp121 [14360.675902] LNetError: 463671:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.126 [14360.711794] LNetError: 463671:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [14360.730854] LNetError: Protocol error connecting to 192.168.202.126@tcp128 at host 192.168.202.126:988: is it running a compatible version of Lustre? [14360.755482] LNetError: Skipped 13 previous similar messages [14362.724202] LNetError: 463679:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.126 [14362.749192] LNetError: 463679:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14362.761969] LNetError: Protocol error connecting to 192.168.202.126@tcp128 at host 192.168.202.126:988: is it running a compatible version of Lustre? [14362.777465] LNetError: Skipped 1 previous similar message [14365.215938] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [14366.820857] LNetError: 463675:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.126 [14366.829653] LNetError: Protocol error connecting to 192.168.202.126@tcp128 at host 192.168.202.126:988: is it running a compatible version of Lustre? [14375.012714] LNetError: 461105:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.126 [14375.031861] LNetError: Protocol error connecting to 192.168.202.126@tcp128 at host 192.168.202.126:988: is it running a compatible version of Lustre? [14375.450904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14375.577576] Lustre: 463665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14375.639969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14375.646417] Lustre: Skipped 1 previous similar message [14375.737895] Lustre: lustre-MDT0001: new disk, initializing [14375.793715] Lustre: 464465:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.126@tcp: can't add NID 192.168.202.126@tcp130: rc = -75 [14375.803634] Lustre: 464465:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14375.932604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14375.958829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14375.973575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14379.356855] Lustre: 464667:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.126@tcp: can't add NID 192.168.202.126@tcp130: rc = -75 [14379.374099] Lustre: 464667:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14380.147708] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [14388.575184] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14389.475753] LNetError: 461104:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.126 [14389.491880] LNetError: Protocol error connecting to 192.168.202.126@tcp106 at host 192.168.202.126:988: is it running a compatible version of Lustre? [14393.930384] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14400.259939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14400.586895] Lustre: lustre-OST0000: new disk, initializing [14400.591454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14400.602993] Lustre: 465638:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14400.677485] Lustre: 465643:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.126@tcp: can't add NID 192.168.202.126@tcp130: rc = -75 [14400.684541] Lustre: 465643:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14400.709413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14405.860491] LNetError: 461106:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.126 [14405.866378] LNetError: 461106:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [14405.874634] LNetError: Protocol error connecting to 192.168.202.126@tcp125 at host 192.168.202.126:988: is it running a compatible version of Lustre? [14405.884242] LNetError: Skipped 18 previous similar messages [14406.777424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14406.786737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14406.958555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14407.084328] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [14416.564648] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14420.070791] LNet: 1 local NIs in recovery (showing 1): 192.168.202.126@tcp128 [14427.618478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14427.623218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14427.631439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14429.935433] Lustre: server umount lustre-OST0000 complete [14437.345418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14437.352079] LustreError: Skipped 1 previous similar message [14437.354663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14437.367499] Lustre: Skipped 1 previous similar message [14437.374962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14437.377741] Lustre: Skipped 1 previous similar message [14439.649063] LNetError: 463671:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.126 [14439.684471] LNetError: 463671:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 27 previous similar messages [14439.707621] LNetError: Protocol error connecting to 192.168.202.126@tcp124 at host 192.168.202.126:988: is it running a compatible version of Lustre? [14439.727761] LNetError: Skipped 27 previous similar messages [14440.247293] Lustre: server umount lustre-MDT0000 complete [14445.897829] LustreError: 465637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788762958 with bad export cookie 6479464012470270746 [14445.901397] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14445.926599] LustreError: 465637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14446.765560] Lustre: server umount lustre-MDT0001 complete [14461.208654] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [14463.883630] Key type lgssc unregistered [14464.402760] LNet: 467550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14464.411278] LNetError: 467550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14464.425910] LNet: Removed LNI 192.168.202.126@tcp [14465.439162] LNet: Removed LNI 192.168.202.126@tcp100 [14465.441456] LNetError: 467550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14465.453522] LNetError: 467550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14465.469973] LNet: Removed LNI 192.168.202.126@tcp101 [14466.527277] LNet: Removed LNI 192.168.202.126@tcp102 [14466.542307] LNetError: 467550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14466.550995] LNetError: 467550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14466.567888] LNet: Removed LNI 192.168.202.126@tcp103 [14467.615161] LNet: Removed LNI 192.168.202.126@tcp104 [14468.639156] LNet: Removed LNI 192.168.202.126@tcp105 [14468.641208] LNetError: 467550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14468.659554] LNetError: 467550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14468.681223] LNet: Removed LNI 192.168.202.126@tcp106 [14469.727091] LNet: Removed LNI 192.168.202.126@tcp107 [14470.754462] LNet: Removed LNI 192.168.202.126@tcp108 [14471.778081] LNet: Removed LNI 192.168.202.126@tcp109 [14472.799189] LNet: Removed LNI 192.168.202.126@tcp110 [14472.803674] LNetError: 467550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14472.808037] LNetError: 467550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14472.812419] LNet: Removed LNI 192.168.202.126@tcp111 [14473.823933] LNet: Removed LNI 192.168.202.126@tcp112 [14474.847226] LNet: Removed LNI 192.168.202.126@tcp113 [14475.872867] LNet: Removed LNI 192.168.202.126@tcp114 [14476.895507] LNet: Removed LNI 192.168.202.126@tcp115 [14477.919236] LNet: Removed LNI 192.168.202.126@tcp116 [14478.943215] LNet: Removed LNI 192.168.202.126@tcp117 [14479.967552] LNet: Removed LNI 192.168.202.126@tcp118 [14480.991542] LNet: Removed LNI 192.168.202.126@tcp119 [14480.993802] LNetError: 467550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14481.008522] LNetError: 467550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14481.028582] LNet: Removed LNI 192.168.202.126@tcp120 [14482.079318] LNet: Removed LNI 192.168.202.126@tcp121 [14483.104422] LNet: Removed LNI 192.168.202.126@tcp122 [14484.129283] LNet: Removed LNI 192.168.202.126@tcp123 [14485.151895] LNet: Removed LNI 192.168.202.126@tcp124 [14486.239784] LNet: Removed LNI 192.168.202.126@tcp125 [14487.264323] LNet: Removed LNI 192.168.202.126@tcp126 [14488.287170] LNet: Removed LNI 192.168.202.126@tcp127 [14488.292936] LNet: Removed LNI 192.168.202.126@tcp128 [14489.313047] LNet: Removed LNI 192.168.202.126@tcp129 [14489.315118] LNet: Removed LNI 192.168.202.126@tcp130 [14489.327403] LNet: Removed LNI 192.168.202.126@tcp131 [14489.337788] LNet: Removed LNI 192.168.202.126@tcp132 [14489.343533] LNet: Removed LNI 192.168.202.126@tcp133 [14489.345394] LNet: Removed LNI 192.168.202.126@tcp134 [14489.399272] LNet: Removed LNI 192.168.202.126@tcp135 [14490.268498] Key type .llcrypt unregistered [14490.269794] Key type ._llcrypt unregistered [14518.210662] Key type ._llcrypt registered [14518.218591] Key type .llcrypt registered [14518.385289] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [14535.821165] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [14537.866092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14537.887651] alg: No test for adler32 (adler32-zlib) [14539.248738] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [14539.796281] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [14541.679219] Key type lgssc registered [14543.147986] Lustre: Echo OBD driver; http://www.lustre.org/ [14581.389843] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 02:38:12 (1788763092) [14599.976625] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [14602.364193] Key type lgssc unregistered [14602.602715] LNet: 471635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14602.616345] LNetError: 471635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14602.635122] LNet: Removed LNI 192.168.202.126@tcp [14603.430375] Key type .llcrypt unregistered [14603.431743] Key type ._llcrypt unregistered [14619.767571] Key type ._llcrypt registered [14619.769447] Key type .llcrypt registered [14619.901072] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [14622.216852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14622.278124] alg: No test for adler32 (adler32-zlib) [14623.951287] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [14624.602798] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [14626.567226] Key type lgssc registered [14628.168352] Lustre: Echo OBD driver; http://www.lustre.org/ [14642.251136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14642.261891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14643.545156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14643.570036] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14643.660579] Lustre: lustre-MDT0000: new disk, initializing [14643.807676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14643.830293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14651.529875] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [14663.514357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14663.732965] Lustre: 472848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14663.795184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14663.798991] Lustre: Skipped 1 previous similar message [14663.937174] Lustre: lustre-MDT0001: new disk, initializing [14664.084414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14664.128159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14664.152229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14669.212990] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [14678.929180] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14685.267841] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14691.978735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14692.394781] Lustre: lustre-OST0000: new disk, initializing [14692.397984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14692.433271] Lustre: 474815:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14692.640720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14700.624746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14700.633923] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14700.831972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14701.214572] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [14711.826696] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14717.347730] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing lnet_if_list [14722.318291] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14722.346222] LNet: Added LNI 192.168.202.126@tcp10 [8/256/0/180] [14728.927062] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14728.941821] LNet: Added LNI 192.168.202.126@tcp11 [8/256/0/180] [14734.342743] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14734.399140] LNet: Added LNI 192.168.202.126@tcp12 [8/256/0/180] [14739.692356] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14739.736763] LNet: Added LNI 192.168.202.126@tcp13 [8/256/0/180] [14744.978263] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14745.069074] LNet: Added LNI 192.168.202.126@tcp14 [8/256/0/180] [14751.013505] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14751.033553] LNet: Added LNI 192.168.202.126@tcp15 [8/256/0/180] [14756.216974] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14756.290847] LNet: Added LNI 192.168.202.126@tcp16 [8/256/0/180] [14761.410875] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14761.436757] LNet: Added LNI 192.168.202.126@tcp17 [8/256/0/180] [14767.056727] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14767.088114] LNet: Added LNI 192.168.202.126@tcp18 [8/256/0/180] [14772.265994] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14772.294768] LNet: Added LNI 192.168.202.126@tcp19 [8/256/0/180] [14778.542696] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14778.574688] LNet: Added LNI 192.168.202.126@tcp20 [8/256/0/180] [14785.839756] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14785.918667] LNet: Added LNI 192.168.202.126@tcp21 [8/256/0/180] [14792.801813] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14792.851612] LNet: Added LNI 192.168.202.126@tcp22 [8/256/0/180] [14799.293492] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14799.336039] LNet: Added LNI 192.168.202.126@tcp23 [8/256/0/180] [14805.870024] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14805.891636] LNet: Added LNI 192.168.202.126@tcp24 [8/256/0/180] [14811.958830] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14812.033487] LNet: Added LNI 192.168.202.126@tcp25 [8/256/0/180] [14818.273430] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14818.295193] LNet: Added LNI 192.168.202.126@tcp26 [8/256/0/180] [14824.195891] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14824.218101] LNet: Added LNI 192.168.202.126@tcp27 [8/256/0/180] [14830.363583] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14830.409478] LNet: Added LNI 192.168.202.126@tcp28 [8/256/0/180] [14837.078231] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14837.106635] LNet: Added LNI 192.168.202.126@tcp29 [8/256/0/180] [14845.406931] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14845.473604] LNet: Added LNI 192.168.202.126@tcp30 [8/256/0/180] [14852.917477] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14852.933824] LNet: Added LNI 192.168.202.126@tcp31 [8/256/0/180] [14862.039730] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14862.100575] LNet: Added LNI 192.168.202.126@tcp32 [8/256/0/180] [14870.434767] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14870.499485] LNet: Added LNI 192.168.202.126@tcp33 [8/256/0/180] [14879.678626] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14879.699946] LNet: Added LNI 192.168.202.126@tcp34 [8/256/0/180] [14887.833912] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14887.907886] LNet: Added LNI 192.168.202.126@tcp35 [8/256/0/180] [14896.876721] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14896.962778] LNet: Added LNI 192.168.202.126@tcp36 [8/256/0/180] [14909.039664] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14909.089737] LNet: Added LNI 192.168.202.126@tcp37 [8/256/0/180] [14919.313861] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14919.342421] LNet: Added LNI 192.168.202.126@tcp38 [8/256/0/180] [14930.874658] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14930.927653] LNet: Added LNI 192.168.202.126@tcp39 [8/256/0/180] [14941.493660] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14941.518358] LNet: Added LNI 192.168.202.126@tcp40 [8/256/0/180] [14951.103829] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14951.129833] LNet: Added LNI 192.168.202.126@tcp41 [8/256/0/180] [14959.530575] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14959.547950] LNet: Added LNI 192.168.202.126@tcp42 [8/256/0/180] [14968.129799] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14968.165314] LNet: Added LNI 192.168.202.126@tcp43 [8/256/0/180] [14976.306472] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14976.330960] LNet: Added LNI 192.168.202.126@tcp44 [8/256/0/180] [14984.527286] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14984.601601] LNet: Added LNI 192.168.202.126@tcp45 [8/256/0/180] [14991.555494] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14991.586702] LNet: Added LNI 192.168.202.126@tcp46 [8/256/0/180] [15002.810547] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15002.849637] LNet: Added LNI 192.168.202.126@tcp47 [8/256/0/180] [15014.225365] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15014.241486] LNet: Added LNI 192.168.202.126@tcp48 [8/256/0/180] [15021.278380] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15021.315836] LNet: Added LNI 192.168.202.126@tcp49 [8/256/0/180] [15030.093208] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15030.139986] LNet: Added LNI 192.168.202.126@tcp50 [8/256/0/180] [15049.184591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15049.190859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15049.206288] Lustre: Skipped 1 previous similar message [15049.218637] Lustre: Skipped 2 previous similar messages [15054.308199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15054.323617] Lustre: Skipped 2 previous similar messages [15055.617728] Lustre: server umount lustre-MDT0000 complete [15059.790185] LustreError: 472841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788763572 with bad export cookie 10185964332188103544 [15059.806493] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15059.812763] LustreError: 472841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15060.169660] Lustre: server umount lustre-MDT0001 complete [15079.904037] Lustre: 472119:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788763576/real 1788763576] req@00000000edf4d51c x1875654094275328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788763592 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15079.979401] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15080.004476] Lustre: Skipped 1 previous similar message [15080.885613] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [15085.024147] Lustre: 472120:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788763581/real 1788763581] req@0000000019768618 x1875654094275456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788763597 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15086.111104] Lustre: 472120:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788763582/real 1788763582] req@00000000e55c170c x1875654094275712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788763598 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15090.817360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15092.192381] Lustre: 472118:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788763588/real 1788763588] req@0000000083e4f5a0 x1875654094275840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788763604 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15105.633350] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8d5bd097b2defdab [15105.647084] Lustre: MGC192.168.202.126@tcp: Connection restored to 0@lo (at 0@lo) [15105.932874] LustreError: 474809:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15105.951188] LustreError: 474809:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [15106.039073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15110.049330] LustreError: 474810:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15110.065964] LustreError: 474810:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [15111.053442] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15114.152445] LustreError: 474809:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15114.164849] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15119.329593] LustreError: 474810:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15120.606111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15121.164847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15125.300977] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15125.955425] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15134.684506] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15141.775949] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15161.325043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15161.349991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15161.352735] Lustre: Skipped 1 previous similar message [15166.434271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15166.447562] Lustre: Skipped 1 previous similar message [15171.553416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15171.568563] Lustre: Skipped 1 previous similar message [15175.135308] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15175.417754] Lustre: server umount lustre-OST0000 complete [15181.793097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15181.812202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15181.816592] Lustre: Skipped 2 previous similar messages [15181.833690] Lustre: Skipped 1 previous similar message [15185.616552] Lustre: server umount lustre-MDT0000 complete [15186.923868] LustreError: 499069:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15186.953616] LustreError: 499069:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [15189.548143] LustreError: 473642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788763701 with bad export cookie 10185964332188106155 [15189.567665] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15189.947883] Lustre: server umount lustre-MDT0001 complete [15202.675284] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [15206.020680] Key type lgssc unregistered [15206.558352] LNet: 501848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15206.572770] LNetError: 501848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15206.583956] LNet: Removed LNI 192.168.202.126@tcp [15206.586526] LNet: Removed LNI 192.168.202.126@tcp10 [15206.589988] LNet: Removed LNI 192.168.202.126@tcp11 [15206.592749] LNet: Removed LNI 192.168.202.126@tcp12 [15206.596438] LNet: Removed LNI 192.168.202.126@tcp13 [15206.599509] LNet: Removed LNI 192.168.202.126@tcp14 [15206.603284] LNet: Removed LNI 192.168.202.126@tcp15 [15206.608376] LNet: Removed LNI 192.168.202.126@tcp16 [15206.613129] LNet: Removed LNI 192.168.202.126@tcp17 [15206.616324] LNet: Removed LNI 192.168.202.126@tcp18 [15206.618553] LNet: Removed LNI 192.168.202.126@tcp19 [15206.621210] LNet: Removed LNI 192.168.202.126@tcp20 [15206.623983] LNet: Removed LNI 192.168.202.126@tcp21 [15206.627144] LNet: Removed LNI 192.168.202.126@tcp22 [15206.629743] LNet: Removed LNI 192.168.202.126@tcp23 [15206.633972] LNet: Removed LNI 192.168.202.126@tcp24 [15206.635901] LNet: Removed LNI 192.168.202.126@tcp25 [15206.638665] LNet: Removed LNI 192.168.202.126@tcp26 [15206.641455] LNet: Removed LNI 192.168.202.126@tcp27 [15206.643538] LNet: Removed LNI 192.168.202.126@tcp28 [15206.645886] LNet: Removed LNI 192.168.202.126@tcp29 [15206.657681] LNet: Removed LNI 192.168.202.126@tcp30 [15206.664355] LNet: Removed LNI 192.168.202.126@tcp31 [15206.668483] LNet: Removed LNI 192.168.202.126@tcp32 [15206.678325] LNet: Removed LNI 192.168.202.126@tcp33 [15206.680258] LNet: Removed LNI 192.168.202.126@tcp34 [15206.683715] LNet: Removed LNI 192.168.202.126@tcp35 [15206.701151] LNet: Removed LNI 192.168.202.126@tcp36 [15206.705244] LNet: Removed LNI 192.168.202.126@tcp37 [15206.709108] LNet: Removed LNI 192.168.202.126@tcp38 [15206.712337] LNet: Removed LNI 192.168.202.126@tcp39 [15206.726991] LNet: Removed LNI 192.168.202.126@tcp40 [15206.728904] LNet: Removed LNI 192.168.202.126@tcp41 [15206.736900] LNet: Removed LNI 192.168.202.126@tcp42 [15206.742843] LNet: Removed LNI 192.168.202.126@tcp43 [15206.752182] LNet: Removed LNI 192.168.202.126@tcp44 [15206.757527] LNet: Removed LNI 192.168.202.126@tcp45 [15206.759550] LNet: Removed LNI 192.168.202.126@tcp46 [15206.763871] LNet: Removed LNI 192.168.202.126@tcp47 [15206.767261] LNet: Removed LNI 192.168.202.126@tcp48 [15206.772408] LNet: Removed LNI 192.168.202.126@tcp49 [15206.782216] LNet: Removed LNI 192.168.202.126@tcp50 [15207.701416] Key type .llcrypt unregistered [15207.703464] Key type ._llcrypt unregistered [15232.121213] Key type ._llcrypt registered [15232.129970] Key type .llcrypt registered [15232.264591] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [15251.809292] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [15253.542345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15253.594413] alg: No test for adler32 (adler32-zlib) [15254.672858] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [15255.067469] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [15256.927404] Key type lgssc registered [15258.300753] Lustre: Echo OBD driver; http://www.lustre.org/ [15298.425701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15298.461720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15299.944287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15300.017888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15300.168076] Lustre: lustre-MDT0000: new disk, initializing [15300.299995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15300.340960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15304.747438] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15318.474145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15318.748438] Lustre: 505043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15318.806772] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15318.810206] Lustre: Skipped 1 previous similar message [15318.868594] Lustre: lustre-MDT0001: new disk, initializing [15318.960992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15318.994654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15319.004974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15324.424720] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15335.230243] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15342.236799] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15350.763215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15351.031800] Lustre: *** cfs_fail_loc=90f, val=0*** [15351.158424] Lustre: lustre-OST0000: new disk, initializing [15351.167591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15351.189429] Lustre: 507061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15351.331922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15358.969129] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15359.105789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15359.127767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15359.219903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15369.538867] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15379.945958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15379.947558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15379.973578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15379.977487] LustreError: Skipped 1 previous similar message [15379.990048] Lustre: Skipped 1 previous similar message [15383.524355] Lustre: server umount lustre-OST0000 complete [15390.179817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15390.206584] Lustre: Skipped 1 previous similar message [15390.225478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15393.503628] Lustre: server umount lustre-MDT0000 complete [15396.940380] LustreError: 505036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788763909 with bad export cookie 3044238145658340543 [15396.959268] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15396.964510] LustreError: 505036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15397.357884] Lustre: server umount lustre-MDT0001 complete [15411.252738] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [15415.423541] Key type lgssc unregistered [15415.856551] LNet: 508966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15415.861288] LNetError: 508966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15415.908342] LNet: Removed LNI 192.168.202.126@tcp [15417.355562] Key type .llcrypt unregistered [15417.359060] Key type ._llcrypt unregistered [15437.688620] Key type ._llcrypt registered [15437.693312] Key type .llcrypt registered [15437.825223] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [15439.052574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15439.149078] alg: No test for adler32 (adler32-zlib) [15440.273446] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [15440.625848] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [15442.447215] Key type lgssc registered [15443.531462] Lustre: Echo OBD driver; http://www.lustre.org/ [15450.943105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15450.980996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15452.557409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15457.696945] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15461.861148] LustreError: 510233:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15461.886869] LustreError: 510233:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [15466.697983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15467.049713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15471.917667] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15479.669882] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15485.585435] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15492.573838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15492.911118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15499.033651] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15507.252889] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15521.977287] loop0: detected capacity change from 0 to 204800000 [15522.249943] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15522.268879] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15522.294669] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15528.691041] loop0: detected capacity change from 0 to 204800000 [15528.998076] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15529.010253] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15529.022364] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15534.552518] loop0: detected capacity change from 0 to 204800000 [15541.302743] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15541.531296] Lustre: fs15246-OST0000: new disk, initializing [15541.534826] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15541.546638] Lustre: 513829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15541.628585] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15549.661299] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15559.501368] loop1: detected capacity change from 0 to 204800000 [15565.602268] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15565.766240] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15565.817073] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15565.889933] Lustre: fs15246-MDT0000: new disk, initializing [15566.128893] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15566.201909] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15566.313300] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15567.396676] Lustre: 514951:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000066b4253c x1875654949781248/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788764095 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15571.423922] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15576.486769] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15576.495955] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15576.530378] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15594.977889] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15595.021295] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15600.119966] LustreError: 515329:0:(ldlm_lib.c:1202:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15600.379331] Lustre: server umount fs15246-MDT0000 complete [15605.224861] LustreError: 515329:0:(ldlm_lib.c:1202:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15610.348399] LustreError: 510230:0:(ldlm_lib.c:1202:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15610.488188] Lustre: server umount fs15246-OST0000 complete [15625.184819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15625.188319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15625.189061] LustreError: Skipped 1 previous similar message [15625.208100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15629.277522] Lustre: server umount lustre-OST0000 complete [15635.939787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15635.940708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15635.944966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15635.944975] Lustre: Skipped 1 previous similar message [15635.950173] Lustre: Skipped 2 previous similar messages [15641.058598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15641.071032] Lustre: Skipped 2 previous similar messages [15646.182109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15646.193988] Lustre: Skipped 1 previous similar message [15648.224816] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15648.511991] Lustre: server umount lustre-MDT0000 complete [15651.299734] LustreError: 510234:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15651.308485] LustreError: 510234:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [15652.621091] LustreError: 510214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788764164 with bad export cookie 5039162884386699537 [15652.632293] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15653.111642] Lustre: server umount lustre-MDT0001 complete [15663.474120] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [15665.891551] Key type lgssc unregistered [15666.084561] LNet: 518048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15666.090744] LNetError: 518048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15666.103339] LNet: Removed LNI 192.168.202.126@tcp [15666.909910] Key type .llcrypt unregistered [15666.911685] Key type ._llcrypt unregistered [15692.273500] Key type ._llcrypt registered [15692.282412] Key type .llcrypt registered [15692.432682] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [15693.706601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15693.750098] alg: No test for adler32 (adler32-zlib) [15695.175149] Lustre: Lustre: Build Version: 2.17.58_41_g2290de3 [15695.449357] LNet: Added LNI 192.168.202.126@tcp [8/256/0/180] [15697.151246] Key type lgssc registered [15698.401551] Lustre: Echo OBD driver; http://www.lustre.org/ [15737.538996] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [15750.707959] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [15788.386563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15788.428842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15789.723983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15789.774931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15789.835267] Lustre: lustre-MDT0000: new disk, initializing [15789.927848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15789.953230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15795.182993] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15806.612512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15806.821717] Lustre: 522514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15806.894861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15806.898469] Lustre: Skipped 1 previous similar message [15807.015282] Lustre: lustre-MDT0001: new disk, initializing [15807.130816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15807.168372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15807.183145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15812.280803] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15822.010835] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15828.670980] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15835.528615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15835.745132] Lustre: lustre-OST0000: new disk, initializing [15835.746996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15835.752438] Lustre: 524483:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15835.830907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15842.542156] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all [15844.963624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15844.979933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15845.077386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15853.243376] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15859.112453] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15859.374665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15865.012116] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15865.386978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15870.953314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15870.963373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15870.976241] Lustre: Skipped 1 previous similar message [15870.979617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15870.984861] LustreError: Skipped 1 previous similar message [15871.009596] Lustre: Skipped 1 previous similar message [15874.010648] Lustre: server umount lustre-OST0000 complete [15879.151085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15879.159205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15879.187208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15881.189061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15881.189793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15884.746453] Lustre: server umount lustre-MDT0000 complete [15888.665240] LustreError: 522505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788764400 with bad export cookie 16503974677026033167 [15888.675074] LustreError: MGC192.168.202.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15888.680958] LustreError: 522505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15889.296428] Lustre: server umount lustre-MDT0001 complete [15914.989406] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid [15926.736744] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local [15962.223374] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15707 sec ========= 03:01:13 (1788764473) [15964.184139] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:01:15 (1788764475) === [15968.319813] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:01:19 (1788764479) === [16011.595341] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing unload_modules_local [16015.045922] Key type lgssc unregistered [16015.299124] LNet: 531143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16015.315959] LNetError: 531143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16015.352909] LNet: Removed LNI 192.168.202.126@tcp [16016.582716] Key type .llcrypt unregistered [16016.584589] Key type ._llcrypt unregistered