[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 512717678 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002335] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007545] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010117] LSM: Security Framework initializing [ 0.011041] Yama: becoming mindful. [ 0.012027] SELinux: Initializing. [ 0.013055] *** VALIDATE selinux *** [ 0.020731] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024617] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026077] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027092] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028090] *** VALIDATE tmpfs *** [ 0.030090] *** VALIDATE proc *** [ 0.031212] *** VALIDATE cgroup *** [ 0.032008] *** VALIDATE cgroup2 *** [ 0.034076] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035132] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039021] Spectre V2 : User space: Vulnerable [ 0.040006] Speculative Store Bypass: Vulnerable [ 0.043244] debug: unmapping init [mem 0xffffffff87c59000-0xffffffff87c60fff] [ 0.045239] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046608] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047019] ... version: 2 [ 0.048010] ... bit width: 48 [ 0.049009] ... generic registers: 4 [ 0.050009] ... value mask: 0000ffffffffffff [ 0.051010] ... max period: 00007fffffffffff [ 0.052009] ... fixed-purpose events: 3 [ 0.053007] ... event mask: 000000070000000f [ 0.054300] rcu: Hierarchical SRCU implementation. [ 0.056472] smp: Bringing up secondary CPUs ... [ 0.057475] x86: Booting SMP configuration: [ 0.058018] .... node #0, CPUs: #1 #2 #3 [ 0.063016] smp: Brought up 1 node, 4 CPUs [ 0.065015] smpboot: Max logical packages: 1 [ 0.066010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.210192] node 0 deferred pages initialised in 143ms [ 0.215065] devtmpfs: initialized [ 0.216412] x86/mm: Memory block size: 128MB [ 0.219719] gcov: version magic: 0x41383552 [ 0.222491] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.227253] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.231325] pinctrl core: initialized pinctrl subsystem [ 0.234179] [ 0.234947] ************************************************************* [ 0.237014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.240013] ** ** [ 0.242012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.245014] ** ** [ 0.248011] ** This means that this kernel is built to expose internal ** [ 0.251012] ** IOMMU data structures, which may compromise security on ** [ 0.254013] ** your system. ** [ 0.256010] ** ** [ 0.258015] ** If you see this message and you are not debugging the ** [ 0.261014] ** kernel, report this immediately to your vendor! ** [ 0.264014] ** ** [ 0.267013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.269012] ************************************************************* [ 0.272686] NET: Registered protocol family 16 [ 0.275434] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.279061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.282073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.287016] cpuidle: using governor menu [ 0.288589] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.290652] PCI: Using configuration type 1 for base access [ 0.293229] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.305074] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.307019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.311133] cryptd: max_cpu_qlen set to 1000 [ 0.313131] ACPI: Added _OSI(Module Device) [ 0.314012] ACPI: Added _OSI(Processor Device) [ 0.316039] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.318014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.322653] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.328632] ACPI: Interpreter enabled [ 0.330056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.331009] ACPI: Using IOAPIC for interrupt routing [ 0.333085] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.336461] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.347981] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.350035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.352014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.355071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.359474] acpiphp: Slot [2] registered [ 0.361179] acpiphp: Slot [5] registered [ 0.362112] acpiphp: Slot [6] registered [ 0.364091] acpiphp: Slot [7] registered [ 0.365103] acpiphp: Slot [8] registered [ 0.366111] acpiphp: Slot [9] registered [ 0.368121] acpiphp: Slot [10] registered [ 0.369103] acpiphp: Slot [3] registered [ 0.371081] acpiphp: Slot [4] registered [ 0.372085] acpiphp: Slot [11] registered [ 0.373108] acpiphp: Slot [12] registered [ 0.374078] acpiphp: Slot [13] registered [ 0.376080] acpiphp: Slot [14] registered [ 0.377094] acpiphp: Slot [15] registered [ 0.378081] acpiphp: Slot [16] registered [ 0.380127] acpiphp: Slot [17] registered [ 0.381097] acpiphp: Slot [18] registered [ 0.383087] acpiphp: Slot [19] registered [ 0.384105] acpiphp: Slot [20] registered [ 0.386101] acpiphp: Slot [21] registered [ 0.387096] acpiphp: Slot [22] registered [ 0.388090] acpiphp: Slot [23] registered [ 0.390091] acpiphp: Slot [24] registered [ 0.391091] acpiphp: Slot [25] registered [ 0.393170] acpiphp: Slot [26] registered [ 0.394136] acpiphp: Slot [27] registered [ 0.396089] acpiphp: Slot [28] registered [ 0.397079] acpiphp: Slot [29] registered [ 0.398063] acpiphp: Slot [30] registered [ 0.400108] acpiphp: Slot [31] registered [ 0.402091] PCI host bridge to bus 0000:00 [ 0.404019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.406020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.409020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.412022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.415023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.417020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.419226] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.422083] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.425449] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.436741] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.440007] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.443017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.446017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.449016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.450000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.453938] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.456040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.461081] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.467017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.482015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.489013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.494965] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.501016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.510020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.525014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.534000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.546013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.558016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.584014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.596279] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.607015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.618015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.641015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.659350] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.670014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.682020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.714015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.729665] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.740014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.745013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.766014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.779998] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.789014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.796014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.816017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.831960] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.834460] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.836345] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.839407] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.842275] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.846164] iommu: Default domain type: Passthrough [ 0.848416] SCSI subsystem initialized [ 0.849189] ACPI: bus type USB registered [ 0.851106] usbcore: registered new interface driver usbfs [ 0.852173] usbcore: registered new interface driver hub [ 0.854068] usbcore: registered new device driver usb [ 0.855131] pps_core: LinuxPPS API ver. 1 registered [ 0.857012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.861077] PTP clock support registered [ 0.863054] EDAC MC: Ver: 3.0.0 [ 0.864161] PCI: Using ACPI for IRQ routing [ 0.865031] NetLabel: Initializing [ 0.866008] NetLabel: domain hash size = 128 [ 0.867011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.869076] NetLabel: unlabeled traffic allowed by default [ 0.871272] vgaarb: loaded [ 0.872411] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.873009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.878947] clocksource: Switched to clocksource kvm-clock [ 0.995839] VFS: Disk quotas dquot_6.6.0 [ 0.997572] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.999987] *** VALIDATE ramfs *** [ 1.000922] *** VALIDATE hugetlbfs *** [ 1.002379] pnp: PnP ACPI init [ 1.004931] pnp: PnP ACPI: found 6 devices [ 1.021047] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.024625] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.027079] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.029317] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.031491] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.033536] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.036682] NET: Registered protocol family 2 [ 1.039188] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.043643] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.047195] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.051716] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.055526] TCP: Hash tables configured (established 65536 bind 65536) [ 1.058129] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.061895] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.065760] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.069379] NET: Registered protocol family 1 [ 1.072760] RPC: Registered named UNIX socket transport module. [ 1.075152] RPC: Registered udp transport module. [ 1.076987] RPC: Registered tcp transport module. [ 1.078775] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.080794] NET: Registered protocol family 44 [ 1.082131] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.083390] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.084574] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.086111] PCI: CLS 0 bytes, default 64 [ 1.087632] Unpacking initramfs... [ 2.525172] debug: unmapping init [mem 0xffff8a973cc54000-0xffff8a973ffbffff] [ 2.529577] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.532133] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.535541] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.031150] Initialise system trusted keyrings [ 3.032884] Key type blacklist registered [ 3.035461] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.044926] zbud: loaded [ 3.047910] *** VALIDATE nfs *** [ 3.049194] *** VALIDATE nfs4 *** [ 3.050497] pstore: using deflate compression [ 3.054081] Platform Keyring initialized [ 3.149644] NET: Registered protocol family 38 [ 3.151933] Key type asymmetric registered [ 3.153875] Asymmetric key parser 'x509' registered [ 3.157435] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.162892] io scheduler mq-deadline registered [ 3.165903] io scheduler kyber registered [ 3.168676] io scheduler bfq registered [ 3.171589] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.175973] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.180264] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.183644] ACPI: Power Button [PWRF] [ 3.189199] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.195776] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.208464] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.217234] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.233983] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.262285] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.293100] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.298090] Non-volatile memory driver v1.3 [ 3.300154] Linux agpgart interface v0.103 [ 3.334606] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.338709] vda: detected capacity change from 0 to 76677120 [ 3.356534] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.359585] vdb: detected capacity change from 0 to 1073741824 [ 3.372195] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.375475] vdc: detected capacity change from 0 to 2621440000 [ 3.390498] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.393101] vdd: detected capacity change from 0 to 2621440000 [ 3.406253] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.408988] vde: detected capacity change from 0 to 4294967296 [ 3.423352] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.426259] vdf: detected capacity change from 0 to 4294967296 [ 3.438310] libphy: Fixed MDIO Bus: probed [ 3.453059] usbcore: registered new interface driver usbserial_generic [ 3.457927] usbserial: USB Serial support registered for generic [ 3.465129] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.473896] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.477334] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.482481] mousedev: PS/2 mouse device common for all mice [ 3.487790] rtc_cmos 00:05: RTC can wake from S4 [ 3.495040] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.495493] rtc_cmos 00:05: registered as rtc0 [ 3.508273] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.510868] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.518082] intel_pstate: CPU model not supported [ 3.522874] hid: raw HID events driver (C) Jiri Kosina [ 3.524420] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.529509] usbcore: registered new interface driver usbhid [ 3.529519] usbhid: USB HID core driver [ 3.529679] drop_monitor: Initializing network drop monitor service [ 3.529822] Initializing XFRM netlink socket [ 3.530246] NET: Registered protocol family 10 [ 3.531689] Segment Routing with IPv6 [ 3.531725] NET: Registered protocol family 17 [ 3.537109] mpls_gso: MPLS GSO support [ 3.559248] RAS: Correctable Errors collector initialized. [ 3.561917] AVX version of gcm_enc/dec engaged. [ 3.564611] AES CTR mode by8 optimization enabled [ 3.657097] sched_clock: Marking stable (3657065732, 0)->(4532855642, -875789910) [ 3.663522] registered taskstats version 1 [ 3.667280] Loading compiled-in X.509 certificates [ 3.669520] zswap: loaded using pool lzo/zbud [ 3.697515] Key type big_key registered [ 3.714304] Key type encrypted registered [ 3.717260] ima: No TPM chip found, activating TPM-bypass! [ 3.723653] ima: Allocated hash algorithm: sha1 [ 3.725420] ima: No architecture policies found [ 3.727600] evm: Initialising EVM extended attributes: [ 3.729534] evm: security.selinux [ 3.730614] evm: security.ima [ 3.731947] evm: security.capability [ 3.733390] evm: HMAC attrs: 0x1 [ 3.735920] rtc_cmos 00:05: setting system clock to 2026-09-05 12:42:39 UTC (1788612159) [ 3.741754] debug: unmapping init [mem 0xffffffff88c03000-0xffffffff88dfffff] [ 3.744444] debug: unmapping init [mem 0xffffffff87982000-0xffffffff87c58fff] [ 3.754256] Write protecting the kernel read-only data: 28672k [ 3.758532] debug: unmapping init [mem 0xffffffff86003000-0xffffffff861fffff] [ 3.760864] debug: unmapping init [mem 0xffffffff86914000-0xffffffff869fffff] [ 3.803175] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.810861] systemd[1]: Detected virtualization kvm. [ 3.813629] systemd[1]: Detected architecture x86-64. [ 3.816373] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.848095] systemd[1]: No hostname configured. [ 3.849975] systemd[1]: Set hostname to . [ 3.852383] random: systemd: uninitialized urandom read (16 bytes read) [ 3.854697] systemd[1]: Initializing machine ID from random generator. [ 3.916058] random: ln: uninitialized urandom read (6 bytes read) [ 4.025042] random: systemd: uninitialized urandom read (16 bytes read) [ 4.029864] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.037377] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.049226] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.829440] device-mapper: uevent: version 1.0.3 [ 4.832221] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.712342] scsi host0: ata_piix [ 5.714618] random: fast init done [ 5.724082] scsi host1: ata_piix [ 5.725945] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.728841] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 5.745503] virtio_net virtio0 ens2: renamed from eth0 [ 9.539892] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.239369] random: crng init done [ 10.241037] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.951823] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.435948] printk: systemd: 24 output lines suppressed due to ratelimiting [ 12.755673] SELinux: Disabled at runtime. [ 12.825767] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.835944] systemd[1]: Detected virtualization kvm. [ 12.837831] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.394211] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.398408] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.403789] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.408272] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.412741] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.420289] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.426049] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 13.572369] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.972813] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.339216] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.372042] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.525226] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.540525] EDAC sbridge: Ver: 1.1.2 [ 16.578832] Key type dns_resolver registered [ 16.927944] NFS: Registering the id_resolver key type [ 16.930616] Key type id_resolver registered [ 16.931971] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg319-server login: [ 61.293307] libcfs: loading out-of-tree module taints kernel. [ 61.332795] Key type ._llcrypt registered [ 61.335372] Key type .llcrypt registered [ 61.421240] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 83.547611] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 84.992728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 85.009699] alg: No test for adler32 (adler32-zlib) [ 86.326237] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 87.052543] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 88.855202] Key type lgssc registered [ 90.291141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 107.624316] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 143.174954] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 154.162286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 154.194503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.414786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 155.443852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 155.526268] Lustre: lustre-MDT0000: new disk, initializing [ 155.607860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.623871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 159.230875] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.180279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.253901] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 171.282449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 171.286320] Lustre: Skipped 1 previous similar message [ 171.375184] Lustre: lustre-MDT0001: new disk, initializing [ 171.451924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 171.485615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 171.494294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 175.448330] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.543716] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 187.624921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 187.853456] Lustre: lustre-OST0000: new disk, initializing [ 187.855487] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 187.859106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 187.867406] Lustre: 8449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 187.915125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.061596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 193.067360] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 193.093652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 193.483814] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.363367] hrtimer: interrupt took 6356488 ns [ 208.968173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.104429] Lustre: lustre-OST0001: new disk, initializing [ 209.107934] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.113541] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.169337] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.055195] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.093105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 217.102275] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 217.165974] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 230.061915] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 235.572342] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 242.886870] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing check_logdir /tmp/testlogs/ [ 248.627703] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing yml_node [ 254.753585] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 257.147397] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 259.774524] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 261.292961] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 08:46:54 EDT 2026 [ 276.667213] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 277.852563] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 293.859068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.868703] Lustre: Skipped 3 previous similar messages [ 293.871459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 293.879368] Lustre: Skipped 3 previous similar messages [ 298.980072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 298.985228] Lustre: Skipped 3 previous similar messages [ 299.357755] Lustre: server umount lustre-MDT0000 complete [ 302.913761] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788612458 with bad export cookie 16553810789247460856 [ 302.918760] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 302.929164] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 304.097027] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.115451] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 304.127737] Lustre: Skipped 1 previous similar message [ 309.277774] Lustre: server umount lustre-MDT0001 complete [ 313.237326] Lustre: server umount lustre-OST0000 complete [ 316.931304] Lustre: server umount lustre-OST0001 complete [ 322.900293] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 331.529709] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 367.281606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.676182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 367.744380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 367.903040] Lustre: lustre-MDT0000: new disk, initializing [ 368.001135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.025428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 372.239313] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 383.803747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.887528] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 383.962931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 383.979490] Lustre: Skipped 1 previous similar message [ 384.070258] Lustre: lustre-MDT0001: new disk, initializing [ 384.166654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 384.225408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 384.241447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 388.793233] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 398.063783] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 404.378791] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 412.330061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 412.725377] Lustre: lustre-OST0000: new disk, initializing [ 412.728972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 412.756270] Lustre: 17043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 412.906576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.045134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 414.072909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 414.283065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 420.001770] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 429.434202] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 434.294024] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 434.565822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 439.491240] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 439.781876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 444.915490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 444.919682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.925036] LustreError: Skipped 1 previous similar message [ 444.925854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 444.925860] Lustre: Skipped 2 previous similar messages [ 444.970095] Lustre: Skipped 2 previous similar messages [ 447.825444] Lustre: server umount lustre-OST0000 complete [ 451.040209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 451.049663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.143347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.152556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 455.178204] Lustre: Skipped 3 previous similar messages [ 465.375184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 465.742140] Lustre: server umount lustre-MDT0000 complete [ 469.480367] LustreError: 16237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 469.502807] LustreError: 16237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 469.518275] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788612625 with bad export cookie 16553810789247462389 [ 469.521829] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 469.525421] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 469.865800] Lustre: server umount lustre-MDT0001 complete [ 473.940848] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:50:27 (1788612627) [ 476.445948] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 478.119413] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:50:31 (1788612631) [ 484.567322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.106602] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.181463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.182469] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 490.467544] LustreError: 19455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.563150] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.132022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.944715] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 511.746762] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 518.638571] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 525.868560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 532.746449] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 541.577450] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 554.983132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 554.985500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.987541] LustreError: Skipped 1 previous similar message [ 555.002560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 555.011271] Lustre: Skipped 5 previous similar messages [ 557.827456] Lustre: server umount lustre-OST0000 complete [ 568.273462] Lustre: server umount lustre-MDT0000 complete [ 570.337952] LustreError: 19455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.365429] LustreError: 19455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 572.635849] LustreError: 19434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788612728 with bad export cookie 16553810789247463271 [ 572.643172] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.651578] LustreError: 19434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 572.931327] Lustre: server umount lustre-MDT0001 complete [ 583.979738] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 587.643387] Key type lgssc unregistered [ 587.985173] LNet: 22988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.006752] LNetError: 22988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.028365] LNet: Removed LNI 192.168.203.119@tcp [ 588.818137] Key type .llcrypt unregistered [ 588.822955] Key type ._llcrypt unregistered [ 611.657672] Key type ._llcrypt registered [ 611.660626] Key type .llcrypt registered [ 611.766741] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 613.054584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 613.089666] alg: No test for adler32 (adler32-zlib) [ 614.103188] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 614.284828] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 615.999274] Key type lgssc registered [ 617.119687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 624.727242] loop: module loaded [ 624.814430] loop0: detected capacity change from 0 to 204800000 [ 625.090079] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 625.104123] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 625.410035] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 631.153447] loop0: detected capacity change from 0 to 204800000 [ 631.332935] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 631.349947] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 631.358592] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 636.823667] loop0: detected capacity change from 0 to 204800000 [ 642.474394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 642.519342] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.851108] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 643.920836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 644.059258] Lustre: lustre-MDT0000: new disk, initializing [ 644.138946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 644.169168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 648.377724] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 657.473222] loop1: detected capacity change from 0 to 204800000 [ 663.663760] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 664.067686] Lustre: lustre-OST07c6: new disk, initializing [ 664.071523] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 664.076851] Lustre: Skipped 1 previous similar message [ 664.083419] Lustre: 26168:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 664.168937] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 671.069723] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 671.844686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 671.858977] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 671.895871] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 675.856756] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 675.864817] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 675.869727] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 675.875726] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 675.881981] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 675.887464] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 694.499036] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 694.515303] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 694.522595] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 694.529171] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 694.544891] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 694.556570] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 699.456359] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 699.473967] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 699.480982] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 699.498329] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.515761] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 699.525919] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.533423] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 699.545124] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.554190] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 699.562878] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.570811] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 699.582578] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.782292] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 702.793520] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 702.807467] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 702.816744] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.827452] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 702.831947] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.836742] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 702.845275] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.851194] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 702.856924] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.865884] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 702.877168] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.286366] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 707.295178] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 707.304466] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 707.310913] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.320612] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 707.330885] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.347232] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 707.358204] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.364136] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 707.374067] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.385615] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 707.395018] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.832417] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 718.846499] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 718.862501] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 718.872432] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.886679] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 718.902445] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.919996] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 718.939145] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.955489] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 718.965179] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.971927] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 718.977272] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 768.993758] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 768.995519] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.022685] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 774.113976] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 779.232793] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 780.255130] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 780.354834] Lustre: server umount lustre-OST07c6 complete [ 788.447087] Lustre: server umount lustre-MDT0000 complete [ 803.485350] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 806.138226] Key type lgssc unregistered [ 806.415939] LNet: 28730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.425134] LNetError: 28730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.463573] LNet: Removed LNI 192.168.203.119@tcp [ 807.440309] Key type .llcrypt unregistered [ 807.443802] Key type ._llcrypt unregistered [ 826.108826] Key type ._llcrypt registered [ 826.110576] Key type .llcrypt registered [ 826.249259] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 827.288563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.375823] alg: No test for adler32 (adler32-zlib) [ 828.469080] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 828.758293] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 830.439370] Key type lgssc registered [ 831.557876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 838.849448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 838.879689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.377613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 845.377152] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 850.930614] LustreError: 29998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 850.954484] LustreError: 29998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 854.730586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.346179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 861.265169] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 868.064571] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 873.078736] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 880.813700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 881.227600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 885.386522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 888.123067] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 895.099306] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 915.435274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.465314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 915.466450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 915.481588] Lustre: Skipped 1 previous similar message [ 916.202737] Lustre: server umount lustre-OST0000 complete [ 920.547120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.552137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 920.575644] Lustre: Skipped 2 previous similar messages [ 925.665892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 925.687271] Lustre: Skipped 2 previous similar messages [ 926.843446] Lustre: server umount lustre-MDT0000 complete [ 930.790907] LustreError: 29998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 930.811623] LustreError: 29998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 931.128948] LustreError: 29978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613086 with bad export cookie 9027501861466233317 [ 931.141441] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 931.149238] LustreError: 29978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 931.380776] Lustre: server umount lustre-MDT0001 complete [ 942.995429] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 945.696906] Key type lgssc unregistered [ 946.015674] LNet: 33733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.026414] LNetError: 33733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.050165] LNet: Removed LNI 192.168.203.119@tcp [ 946.895388] Key type .llcrypt unregistered [ 946.898565] Key type ._llcrypt unregistered [ 964.229942] Key type ._llcrypt registered [ 964.236979] Key type .llcrypt registered [ 964.331850] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 965.560453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 965.584810] alg: No test for adler32 (adler32-zlib) [ 966.587810] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 966.881536] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 968.575784] Key type lgssc registered [ 969.664746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 977.099172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 977.126046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.531732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 982.412461] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 988.643336] LustreError: 34999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 988.674298] LustreError: 34999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 989.951762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.274569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 994.320764] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1001.689765] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1007.856220] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1014.931046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1015.354294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.357645] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1023.547337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1027.672279] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1043.936223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1043.945574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1043.959449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1044.963058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1044.969899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1044.975402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1045.026193] Lustre: Skipped 1 previous similar message [ 1048.805681] Lustre: server umount lustre-OST0000 complete [ 1055.201698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.207496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1055.234522] Lustre: Skipped 1 previous similar message [ 1055.275051] Lustre: Skipped 1 previous similar message [ 1059.581158] Lustre: server umount lustre-MDT0000 complete [ 1060.323116] LustreError: 34999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1060.373904] LustreError: 34999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1064.165759] LustreError: 34981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613219 with bad export cookie 12301717190737078169 [ 1064.178142] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1064.187295] LustreError: 34981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1064.567195] Lustre: server umount lustre-MDT0001 complete [ 1077.542738] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1080.726833] Key type lgssc unregistered [ 1081.074979] LNet: 38554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.084940] LNetError: 38554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.118848] LNet: Removed LNI 192.168.203.119@tcp [ 1082.131572] Key type .llcrypt unregistered [ 1082.133394] Key type ._llcrypt unregistered [ 1106.614450] Key type ._llcrypt registered [ 1106.617982] Key type .llcrypt registered [ 1106.832702] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1108.248233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1108.410074] alg: No test for adler32 (adler32-zlib) [ 1109.450063] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 1109.767683] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1111.447889] Key type lgssc registered [ 1112.705604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1120.669103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1120.700870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1122.278823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1127.258675] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1132.518475] LustreError: 39983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1132.535435] LustreError: 39983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1135.490391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1135.982973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1140.529602] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1147.938278] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1153.763895] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1160.798335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1161.318148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1167.517109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1167.933497] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1175.974433] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1186.279578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1186.285131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1186.297266] Lustre: Skipped 2 previous similar messages [ 1186.318449] Lustre: Skipped 2 previous similar messages [ 1189.011213] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 1191.396281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1191.405938] Lustre: Skipped 2 previous similar messages [ 1192.091530] Lustre: server umount lustre-MDT0000 complete [ 1194.101275] LustreError: 39978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.121851] LustreError: 39965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613351 with bad export cookie 3434160268367534232 [ 1196.125576] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1196.134807] LustreError: 39965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1196.519598] LustreError: 42475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.532166] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.567816] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1199.214391] LustreError: 39978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.641692] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1201.653680] Lustre: Skipped 1 previous similar message [ 1202.591501] Lustre: server umount lustre-MDT0001 complete [ 1218.079181] Lustre: server umount lustre-OST0000 complete [ 1234.727815] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1237.313421] Key type lgssc unregistered [ 1237.671787] LNet: 43841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.680121] LNetError: 43841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.702056] LNet: Removed LNI 192.168.203.119@tcp [ 1238.656744] Key type .llcrypt unregistered [ 1238.658911] Key type ._llcrypt unregistered [ 1262.310475] Key type ._llcrypt registered [ 1262.317799] Key type .llcrypt registered [ 1262.464258] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1263.576185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1263.610064] alg: No test for adler32 (adler32-zlib) [ 1264.811654] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 1265.173335] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1266.858260] Key type lgssc registered [ 1268.369736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1276.531710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1276.547642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1278.057476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.970512] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1287.655537] LustreError: 45256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.679516] LustreError: 45256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1291.497268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1292.084725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1296.702403] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1304.495800] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1310.435719] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1316.944821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1317.371336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.471256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1323.628117] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1331.004515] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1341.926250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.932701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1341.941249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1344.138129] Lustre: lustre-OST0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 1344.144162] Lustre: Skipped 1 previous similar message [ 1347.047632] LustreError: 47079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.129287] Lustre: server umount lustre-OST0000 complete [ 1365.025448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1365.027213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1365.031142] Lustre: Skipped 2 previous similar messages [ 1365.042606] Lustre: Skipped 1 previous similar message [ 1371.104975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1371.110151] Lustre: Skipped 3 previous similar messages [ 1371.331754] Lustre: server umount lustre-MDT0000 complete [ 1375.328933] LustreError: 45235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613531 with bad export cookie 7269635222286148611 [ 1375.339269] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1375.345518] LustreError: 45235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1381.949922] Lustre: server umount lustre-MDT0001 complete [ 1392.877508] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1396.552794] Key type lgssc unregistered [ 1396.857701] LNet: 48962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1396.864525] LNetError: 48962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1396.899577] LNet: Removed LNI 192.168.203.119@tcp [ 1397.596727] Key type .llcrypt unregistered [ 1397.601068] Key type ._llcrypt unregistered [ 1419.278132] Key type ._llcrypt registered [ 1419.279762] Key type .llcrypt registered [ 1419.373783] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1419.992558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1420.005395] alg: No test for adler32 (adler32-zlib) [ 1421.026297] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 1421.305605] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1422.951289] Key type lgssc registered [ 1424.360776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1431.668423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1431.686948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1433.171382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1436.974141] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1442.283931] LustreError: 50374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1442.319421] LustreError: 50374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1444.701281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.089485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1448.985346] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1456.032518] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1461.100612] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1467.866178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1468.246127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1475.127423] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1477.512906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1482.055258] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1488.203655] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1491.559308] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1504.161592] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-05 9h07m36s [ 1505.838445] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1508.321738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1508.333204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1508.347479] Lustre: Skipped 2 previous similar messages [ 1508.383895] Lustre: Skipped 2 previous similar messages [ 1509.410178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1509.423615] Lustre: Skipped 2 previous similar messages [ 1510.550717] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 1514.014265] Lustre: server umount lustre-MDT0000 complete [ 1515.491555] LustreError: 50371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1515.514538] LustreError: 50371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1516.659112] LustreError: 50369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1517.666300] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1521.646812] LustreError: 50375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1521.670292] LustreError: 50375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1524.174392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1524.469952] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1524.751037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.817763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1525.817273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1528.868554] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1532.253717] Lustre: DEBUG MARKER: Wait for df (36680) ... [ 1533.862907] Lustre: DEBUG MARKER: done [ 1540.064277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1540.085319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1540.100639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1540.110855] Lustre: Skipped 1 previous similar message [ 1541.090816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1541.101598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1544.396537] Lustre: server umount lustre-OST0000 complete [ 1551.329226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1551.345546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1551.357962] Lustre: Skipped 3 previous similar messages [ 1554.252304] Lustre: server umount lustre-MDT0000 complete [ 1557.476984] LustreError: 50369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1557.493525] LustreError: 50369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1557.541433] LustreError: 52999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613713 with bad export cookie 14667678224783048081 [ 1557.549592] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1557.551474] LustreError: 52999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1557.820949] Lustre: server umount lustre-MDT0001 complete [ 1568.359499] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1570.999811] Key type lgssc unregistered [ 1571.280219] LNet: 55294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.290965] LNetError: 55294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.314877] LNet: Removed LNI 192.168.203.119@tcp [ 1572.115773] Key type .llcrypt unregistered [ 1572.119111] Key type ._llcrypt unregistered [ 1608.947911] Key type ._llcrypt registered [ 1608.951969] Key type .llcrypt registered [ 1609.067451] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1610.074091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1610.107477] alg: No test for adler32 (adler32-zlib) [ 1611.125547] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 1611.319589] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1612.983166] Key type lgssc registered [ 1613.971694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1621.312342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1621.323498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1622.502610] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1622.518348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1622.693416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1626.959312] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1634.173375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.265357] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1634.352052] Lustre: 57499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1634.604875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1639.687323] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1647.838115] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1653.545468] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1660.351626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1660.538606] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1660.727847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1667.234450] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1668.291338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1675.380674] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1681.976060] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-05 9h10m35s [ 1683.741075] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1690.802187] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1693.356272] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1693.431834] LustreError: 57507:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1713.839843] Lustre: lustre-MDT0000: Client bba74b25-6a2f-4106-ae19-037f3158163c (at 192.168.203.19@tcp) reconnecting [ 1733.495342] LustreError: 57507:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1735.553532] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1737.539445] Lustre: DEBUG MARKER: done [ 1745.386976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1745.388989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.395225] LustreError: Skipped 1 previous similar message [ 1745.399206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1745.413866] Lustre: Skipped 1 previous similar message [ 1745.436885] Lustre: Skipped 1 previous similar message [ 1748.095917] Lustre: server umount lustre-OST0000 complete [ 1752.551544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1752.569226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1752.578439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1753.635237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1753.642309] Lustre: Skipped 2 previous similar messages [ 1756.711556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1756.725654] Lustre: Skipped 1 previous similar message [ 1757.656461] Lustre: server umount lustre-MDT0000 complete [ 1761.204602] LustreError: 58240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613916 with bad export cookie 16257255285099349615 [ 1761.217046] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1761.224527] LustreError: 58240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1761.573181] Lustre: server umount lustre-MDT0001 complete [ 1772.059497] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 1774.523827] Key type lgssc unregistered [ 1774.813798] LNet: 61829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.820271] LNetError: 61829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.833791] LNet: Removed LNI 192.168.203.119@tcp [ 1775.574302] Key type .llcrypt unregistered [ 1775.581403] Key type ._llcrypt unregistered [ 1819.002414] Key type ._llcrypt registered [ 1819.004567] Key type .llcrypt registered [ 1819.123298] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 1820.344439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1820.453491] alg: No test for adler32 (adler32-zlib) [ 1821.584288] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 1821.844549] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 1823.575320] Key type lgssc registered [ 1824.699197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1827.994776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1828.037952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1829.387428] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1829.413097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1829.627891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.664965] Lustre: Failing over lustre-MDT0000 [ 1831.080299] Lustre: server umount lustre-MDT0000 complete [ 1837.655528] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:13:10 (1788613990) [ 1844.162032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1844.701021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.063561] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1857.570914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1857.709830] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1857.750527] Lustre: 64623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1858.132846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1863.002751] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1870.570079] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1876.053455] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1882.518260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1882.683664] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1882.913861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1888.908392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1889.475914] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1896.961476] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1902.871390] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1908.884868] Lustre: server umount lustre-MDT0000 complete [ 1909.222658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1909.241057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1912.059791] LustreError: 65359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788614067 with bad export cookie 4619101276738342224 [ 1912.064131] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1912.066258] LustreError: 65359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1912.354468] Lustre: server umount lustre-MDT0001 complete [ 1915.223572] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1919.113202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1928.230760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.167352] Lustre: 63547:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788614070/real 1788614070] req@00000000915dc177 x1875497689584000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788614086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1931.214470] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.226595] Lustre: Skipped 2 previous similar messages [ 1933.279424] Lustre: 63546:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788614073/real 1788614073] req@00000000adf8b15d x1875497689584128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788614089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1935.391116] Lustre: 63546:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788614075/real 1788614075] req@00000000dd9b0095 x1875497689584256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788614091 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1936.770808] LustreError: 66472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1936.802447] LustreError: 66472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1936.941970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1937.058569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1937.887943] Lustre: 63547:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788614077/real 1788614077] req@00000000a9b73b6d x1875497689584512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788614093 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1941.687285] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1941.990377] LustreError: 66473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.007163] Lustre: 63548:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788614082/real 1788614082] req@000000000c3b6645 x1875497689584640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788614098 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1943.013587] LustreError: 68395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.123478] LustreError: 68395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1948.138279] LustreError: 68395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1948.158912] LustreError: 68395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1950.191295] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1950.631880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.086364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1955.785388] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 1956.342851] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1966.258225] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1971.734067] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1978.616312] Lustre: server umount lustre-MDT0000 complete [ 1981.921282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1981.926790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1981.936925] LustreError: 68395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1981.936937] LustreError: 68395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1982.032822] Lustre: Skipped 2 previous similar messages [ 1983.068470] LustreError: 64615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788614138 with bad export cookie 4619101276738344499 [ 1983.073411] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1983.096368] LustreError: 64615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1983.514661] Lustre: server umount lustre-MDT0001 complete [ 1990.041738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1999.633575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.399156] Lustre: 63546:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788614142/real 1788614142] req@00000000b2f65342 x1875497689611392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788614158 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2002.437042] Lustre: 63546:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2002.442716] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.986305] LustreError: 66472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.015647] LustreError: 66472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2008.076860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.159025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2013.295111] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2021.350478] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2022.310170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.611597] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2028.021665] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2035.406980] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2041.073891] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2053.606396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2053.610948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2053.614469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2053.616242] Lustre: Skipped 1 previous similar message [ 2055.663146] Lustre: server umount lustre-MDT0000 complete [ 2058.726760] LustreError: 71076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.763057] LustreError: 71076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 2059.797286] LustreError: 64615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788614215 with bad export cookie 4619101276738345878 [ 2059.799664] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2059.822938] LustreError: 64615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2060.156494] Lustre: server umount lustre-MDT0001 complete [ 2066.693682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2069.816759] Lustre: DEBUG MARKER: files compared the same [ 2082.303801] Lustre: server umount lustre-OST0000 complete [ 2099.581753] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 2102.617630] Key type lgssc unregistered [ 2102.952729] LNet: 74351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2102.965809] LNetError: 74351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.981391] LNet: Removed LNI 192.168.203.119@tcp [ 2104.085216] Key type .llcrypt unregistered [ 2104.087785] Key type ._llcrypt unregistered [ 2126.484565] Key type ._llcrypt registered [ 2126.486882] Key type .llcrypt registered [ 2126.592604] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2127.580829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2127.606671] alg: No test for adler32 (adler32-zlib) [ 2128.675578] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 2128.870980] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 2130.568327] Key type lgssc registered [ 2131.574430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2138.666327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2138.687766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2140.155567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.262912] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug +malloc all [ 2149.351715] LustreError: 75773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.361861] LustreError: 75773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2151.696811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.113661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2158.119395] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug +malloc all [ 2166.869561] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2173.076611] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2180.425435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2180.647996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2184.771852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2188.361973] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug +malloc all [ 2196.742063] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2205.152755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2205.163359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.186076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2207.201328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2207.202547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2207.203279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2207.211102] Lustre: Skipped 1 previous similar message [ 2210.979915] Lustre: server umount lustre-OST0000 complete [ 2217.439741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.444398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2217.449163] Lustre: Skipped 1 previous similar message [ 2222.565226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2222.575170] Lustre: Skipped 2 previous similar messages [ 2227.682238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.693629] Lustre: Skipped 1 previous similar message [ 2229.218631] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2229.353755] Lustre: server umount lustre-MDT0000 complete [ 2232.800216] LustreError: 75774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2232.806724] LustreError: 75774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2232.863584] LustreError: 75755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788614388 with bad export cookie 13822327311425498057 [ 2232.864881] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2232.868891] LustreError: 75755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2233.103965] Lustre: server umount lustre-MDT0001 complete [ 2244.380453] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 2248.281030] Key type lgssc unregistered [ 2248.666669] LNet: 79307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.678016] LNetError: 79307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.729456] LNet: Removed LNI 192.168.203.119@tcp [ 2250.249730] Key type .llcrypt unregistered [ 2250.258180] Key type ._llcrypt unregistered [ 2274.860310] Key type ._llcrypt registered [ 2274.862551] Key type .llcrypt registered [ 2274.963450] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2276.292251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2276.324533] alg: No test for adler32 (adler32-zlib) [ 2277.395936] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 2277.669125] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 2279.351188] Key type lgssc registered [ 2280.414601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2287.143724] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2287.163708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2302.752815] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2307.871382] LustreError: 80695:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.119@tcp: failed processing log, type 4: rc = -110 [ 2334.560741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.201906] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2348.645910] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2356.607614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2356.790989] Lustre: *** cfs_fail_loc=706, val=0*** [ 2357.069372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2357.123296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2362.036605] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2362.346576] LustreError: 80720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.360085] LustreError: 80720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2367.462943] LustreError: 80721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.511883] LustreError: 81779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.530869] LustreError: 81779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2370.987325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.494463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2376.649511] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2383.481851] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2388.813702] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2392.055669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2392.066627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2392.079575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2392.102741] Lustre: Skipped 1 previous similar message [ 2397.154841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2397.173030] Lustre: Skipped 1 previous similar message [ 2402.276880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2402.289685] Lustre: Skipped 2 previous similar messages [ 2404.834338] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2404.986673] Lustre: server umount lustre-OST0000 complete [ 2412.523133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.524787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2412.532864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2412.543307] Lustre: Skipped 1 previous similar message [ 2414.617224] Lustre: server umount lustre-MDT0000 complete [ 2417.643790] LustreError: 81785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.659955] LustreError: 81785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2418.359613] LustreError: 80702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788614574 with bad export cookie 8324793497631932321 [ 2418.369392] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2418.379749] LustreError: 80702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2418.698544] Lustre: server umount lustre-MDT0001 complete [ 2429.147838] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 2431.945845] Key type lgssc unregistered [ 2432.240726] LNet: 84312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2432.259175] LNetError: 84312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2432.287230] LNet: Removed LNI 192.168.203.119@tcp [ 2433.001874] Key type .llcrypt unregistered [ 2433.005577] Key type ._llcrypt unregistered [ 2456.881694] Key type ._llcrypt registered [ 2456.883286] Key type .llcrypt registered [ 2457.025440] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2458.084636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2458.188299] alg: No test for adler32 (adler32-zlib) [ 2459.247754] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 2459.505114] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 2461.223375] Key type lgssc registered [ 2462.368949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2470.181843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2470.207894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.743658] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2483.962316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2484.636241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2489.657798] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2489.825853] LustreError: 86410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.860996] LustreError: 86410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2494.953111] LustreError: 86409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2494.972069] LustreError: 86409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2498.560396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2498.933560] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.962951] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2498.979076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2504.162801] LustreError: 86413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.183757] LustreError: 86413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2505.798232] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2509.280991] LustreError: 86414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.306097] LustreError: 86414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2514.481162] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2514.497161] Lustre: MGS: Client 28136d8a-c3ab-43ed-ba85-4deef82b8abb (at 0@lo) reconnecting [ 2514.525087] LustreError: 86424:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2514.682794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2514.794988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2519.104963] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2535.399035] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.399202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2535.412597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2535.439230] Lustre: Skipped 1 previous similar message [ 2539.676599] Lustre: server umount lustre-OST0000 complete [ 2545.636614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2545.641282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2545.662419] Lustre: Skipped 2 previous similar messages [ 2545.686265] Lustre: Skipped 1 previous similar message [ 2550.013770] Lustre: server umount lustre-MDT0000 complete [ 2550.753025] LustreError: 86413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.787895] LustreError: 86413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2553.881392] Lustre: server umount lustre-MDT0001 complete [ 2557.233096] Lustre: server umount MGS complete [ 2570.111182] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:25:22 (1788614722) [ 2603.552323] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 2614.746460] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2658.702457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.899225] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2675.522309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.810501] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2675.896639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2676.001566] Lustre: lustre-MDT0001: new disk, initializing [ 2676.084584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2676.102965] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2676.110378] Lustre: 93602:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2676.119845] LustreError: 93602:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2681.745368] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2694.765698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2695.003485] Lustre: lustre-OST0000: new disk, initializing [ 2695.007778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2695.013089] Lustre: 94368:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2695.079729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2702.665481] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2703.969799] Lustre: 94731:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000047071c9 x1875498358579328/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788614875 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2713.421693] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2721.958614] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2721.993786] Lustre: MGS: Client 964bb647-6b7b-46e9-a7fd-dd37971d35f2 (at 0@lo) reconnecting [ 2722.085537] Lustre: 92869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2722.128554] LustreError: 93588:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2722.173179] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2722.325837] Lustre: lustre-MDT0000: new disk, initializing [ 2722.544926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2722.579324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2722.605739] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2723.682541] Lustre: 95438:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000091bc5129 x1875498358586240/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788614895 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2728.564071] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2728.864549] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2731.507371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2731.520369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2732.789400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2732.804862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2732.997043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2757.108027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.114276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2757.126944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2757.129365] Lustre: Skipped 1 previous similar message [ 2762.220079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2762.226992] Lustre: Skipped 2 previous similar messages [ 2767.330128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2767.343667] Lustre: Skipped 1 previous similar message [ 2769.375513] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2769.629843] Lustre: server umount lustre-OST0000 complete [ 2777.572908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2777.573247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2777.592428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2780.994179] Lustre: server umount lustre-MDT0000 complete [ 2782.690385] LustreError: 95865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.708169] LustreError: 95865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2791.778432] Lustre: server umount lustre-MDT0001 complete [ 2796.213340] Lustre: server umount MGS complete [ 2808.997903] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:29:22 (1788614962) [ 2830.993924] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 2834.699641] Key type lgssc unregistered [ 2835.181701] LNet: 97940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2835.208743] LNetError: 97940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2835.238956] LNet: Removed LNI 192.168.203.119@tcp [ 2836.420297] Key type .llcrypt unregistered [ 2836.423081] Key type ._llcrypt unregistered [ 2853.218791] Key type ._llcrypt registered [ 2853.221684] Key type .llcrypt registered [ 2853.330951] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2854.309027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2854.317069] alg: No test for adler32 (adler32-zlib) [ 2855.476600] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 2855.665481] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 2857.287200] Key type lgssc registered [ 2858.428489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2866.151121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2866.196144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.829655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.551734] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2877.410554] LustreError: 99158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.445136] LustreError: 99158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2882.190159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.591483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2888.169493] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2895.866282] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2901.024836] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2907.772193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2908.112440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2913.278419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2914.401728] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 2923.076523] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2934.755888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2934.772862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2934.798554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2938.862231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2938.866339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.878076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2940.046303] Lustre: server umount lustre-OST0000 complete [ 2949.094829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.113816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2949.116473] Lustre: Skipped 1 previous similar message [ 2949.120149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2950.338644] Lustre: server umount lustre-MDT0000 complete [ 2954.211314] LustreError: 99157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2954.243461] LustreError: 99157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2954.422243] LustreError: 99138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788615110 with bad export cookie 8999771735502162582 [ 2954.448603] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2955.059492] Lustre: server umount lustre-MDT0001 complete [ 2967.227299] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 2970.820442] Key type lgssc unregistered [ 2971.114219] LNet: 102692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2971.122941] LNetError: 102692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2971.158728] LNet: Removed LNI 192.168.203.119@tcp [ 2972.254734] Key type .llcrypt unregistered [ 2972.259456] Key type ._llcrypt unregistered [ 2990.107271] Key type ._llcrypt registered [ 2990.111857] Key type .llcrypt registered [ 2990.265931] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 2992.046747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2992.081381] alg: No test for adler32 (adler32-zlib) [ 2993.145228] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 2993.500787] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 2995.263346] Key type lgssc registered [ 2996.504786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3003.675524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3003.675694] LustreError: 103772:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3003.682145] Lustre: Skipped 1 previous similar message [ 3003.682168] LustreError: 103773:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3003.711909] LustreError: 103772:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3003.714962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3003.732739] LustreError: 103772:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3003.732764] LustreError: 103772:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3005.265462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.573058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.010748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3019.279834] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3025.519441] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing lsmod [ 3028.543723] LustreError: 104966:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3028.642759] LustreError: 104979:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3028.647871] LustreError: 104966:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4901 [ 3028.654489] LustreError: 104966:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3028.665560] LustreError: 104966:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3028.667676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3029.162451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3030.204632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3034.594949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3034.615058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.650248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3035.620687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3035.626795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3035.664403] Lustre: Skipped 1 previous similar message [ 3038.694602] Lustre: server umount lustre-MDT0000 complete [ 3038.754385] LustreError: 103811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.768808] LustreError: 103811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3042.880102] LustreError: 103784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788615198 with bad export cookie 17977760752410917269 [ 3042.894971] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3042.907940] Lustre: 103803:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.119@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3043.412853] Lustre: server umount lustre-MDT0001 complete [ 3058.260554] Lustre: server umount lustre-OST0000 complete [ 3067.532072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.201331] LustreError: 106042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3068.344858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3073.514534] LustreError: 106043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.123422] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3078.631016] LustreError: 106042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3083.754466] LustreError: 106043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3084.219140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3084.496878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3090.170665] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3098.879671] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3105.347224] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3113.235540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3113.392218] LustreError: 107864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3113.702407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.088242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3121.957136] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3130.309945] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3139.554733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3139.568625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.580664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3139.598839] Lustre: Skipped 1 previous similar message [ 3139.616841] Lustre: Skipped 2 previous similar messages [ 3144.045268] Lustre: server umount lustre-OST0000 complete [ 3149.809897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3149.847376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3149.866549] Lustre: Skipped 1 previous similar message [ 3154.915487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3154.937770] Lustre: Skipped 1 previous similar message [ 3155.802279] Lustre: server umount lustre-MDT0000 complete [ 3160.033684] LustreError: 107008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3160.059750] LustreError: 107008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3160.216480] LustreError: 106023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788615315 with bad export cookie 17977760752410918200 [ 3160.243749] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3160.246504] LustreError: 106023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3160.683892] Lustre: server umount lustre-MDT0001 complete [ 3174.454499] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 3177.928063] Key type lgssc unregistered [ 3178.309072] LNet: 109578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3178.326213] LNetError: 109578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3178.371808] LNet: Removed LNI 192.168.203.119@tcp [ 3179.557757] Key type .llcrypt unregistered [ 3179.561101] Key type ._llcrypt unregistered [ 3211.960512] Key type ._llcrypt registered [ 3211.966693] Key type .llcrypt registered [ 3212.155156] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3213.620299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3213.686217] alg: No test for adler32 (adler32-zlib) [ 3214.969246] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 3215.351721] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 3217.207161] Key type lgssc registered [ 3218.988406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3227.275513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3227.322249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.666971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3234.046220] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3238.890072] LustreError: 110997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.903417] LustreError: 110997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3243.598185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.107219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3249.677093] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3258.821849] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3265.392398] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3273.413055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3273.901125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3274.960282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3280.995948] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3289.588365] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3295.756891] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3299.925478] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3304.928211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3304.938766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.945721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3304.990305] Lustre: Skipped 1 previous similar message [ 3310.051269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3310.065061] Lustre: Skipped 2 previous similar messages [ 3310.633686] Lustre: server umount lustre-OST0000 complete [ 3315.173921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3315.176064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3315.186089] Lustre: Skipped 1 previous similar message [ 3315.205682] Lustre: Skipped 1 previous similar message [ 3320.306896] LustreError: 110997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3320.329514] LustreError: 110997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3320.366331] Lustre: server umount lustre-MDT0000 complete [ 3323.986624] LustreError: 110977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788615479 with bad export cookie 18314230103153353039 [ 3323.987466] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3324.017350] LustreError: 110977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3324.445815] Lustre: server umount lustre-MDT0001 complete [ 3335.559721] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 3337.930914] Key type lgssc unregistered [ 3338.201974] LNet: 114637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3338.208950] LNetError: 114637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3338.225547] LNet: Removed LNI 192.168.203.119@tcp [ 3338.887976] Key type .llcrypt unregistered [ 3338.894118] Key type ._llcrypt unregistered [ 3356.475759] Key type ._llcrypt registered [ 3356.480453] Key type .llcrypt registered [ 3356.582667] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3358.125361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3358.152767] alg: No test for adler32 (adler32-zlib) [ 3359.301771] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 3359.614020] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 3361.327171] Key type lgssc registered [ 3362.457718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3370.268532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3370.294863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3371.771066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3376.190407] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3382.241427] LustreError: 115855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.275349] LustreError: 115855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3385.215854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.655364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3390.548315] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3398.903260] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3405.444723] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3413.540053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3413.982866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3418.129724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3421.196705] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3428.952026] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3438.560210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3438.568460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.589138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3440.610623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3440.619247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.621074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3445.739029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3445.757349] Lustre: Skipped 2 previous similar messages [ 3450.854905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3450.871473] Lustre: Skipped 1 previous similar message [ 3451.359700] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3451.735472] Lustre: server umount lustre-OST0000 complete [ 3455.968903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3455.990127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3455.994678] Lustre: Skipped 1 previous similar message [ 3466.217772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3466.229534] Lustre: Skipped 3 previous similar messages [ 3470.303153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3470.476991] Lustre: server umount lustre-MDT0000 complete [ 3471.335211] LustreError: 117057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3471.364945] LustreError: 117057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3474.343893] LustreError: 115835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788615630 with bad export cookie 3708591686379252623 [ 3474.353621] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3474.365068] LustreError: 115835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3474.855275] Lustre: server umount lustre-MDT0001 complete [ 3487.079404] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 3490.381500] Key type lgssc unregistered [ 3490.808818] LNet: 119391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3490.832585] LNetError: 119391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3490.859455] LNet: Removed LNI 192.168.203.119@tcp [ 3491.849921] Key type .llcrypt unregistered [ 3491.852211] Key type ._llcrypt unregistered [ 3518.959641] Key type ._llcrypt registered [ 3518.962558] Key type .llcrypt registered [ 3519.105766] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3520.579977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3520.649039] alg: No test for adler32 (adler32-zlib) [ 3521.815185] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 3522.105880] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 3523.831197] Key type lgssc registered [ 3524.967551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3533.366410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3533.379923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3534.958067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3540.169030] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3544.548489] LustreError: 120806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3544.563469] LustreError: 120806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3549.172861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3549.868390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3554.845595] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3562.827461] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3568.779632] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3575.556695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3575.943891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.096383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3581.840772] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3589.974670] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3599.921861] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3604.150846] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3604.166315] Lustre: Skipped 2 previous similar messages [ 3606.111288] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3606.156556] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3608.555078] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3608.561808] Lustre: Skipped 2 previous similar messages [ 3615.769037] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3615.773189] Lustre: Skipped 1 previous similar message [ 3619.939675] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.19@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3619.948626] Lustre: Skipped 2 previous similar messages [ 3625.660838] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.19@tcp 192.168.0.[2,10]@tcp [ 3630.042132] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.19/24@tcp in log lustre-MDT0000 [ 3630.054303] Lustre: Skipped 2 previous similar messages [ 3632.671749] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.19/24@tcp [ 3632.689412] Lustre: Skipped 1 previous similar message [ 3640.801653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.804130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3640.810121] Lustre: Skipped 1 previous similar message [ 3640.814476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3640.828792] Lustre: Skipped 1 previous similar message [ 3645.261915] Lustre: server umount lustre-OST0000 complete [ 3651.040269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3651.062582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3651.063740] Lustre: Skipped 1 previous similar message [ 3651.081192] Lustre: Skipped 1 previous similar message [ 3656.164072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.258352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.268555] Lustre: Skipped 2 previous similar messages [ 3663.327581] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3663.530949] Lustre: server umount lustre-MDT0000 complete [ 3665.380903] LustreError: 120807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3665.409432] LustreError: 120807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3666.986880] LustreError: 124196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788615822 with bad export cookie 3210354023832837619 [ 3667.001852] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3667.402407] Lustre: server umount lustre-MDT0001 complete [ 3679.293432] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 3681.949831] Key type lgssc unregistered [ 3682.378066] LNet: 125707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3682.382743] LNetError: 125707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3683.442843] LNet: Removed LNI 192.168.203.119@tcp [ 3684.408768] Key type .llcrypt unregistered [ 3684.415641] Key type ._llcrypt unregistered [ 3715.176116] Key type ._llcrypt registered [ 3715.177616] Key type .llcrypt registered [ 3715.254647] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3716.414904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3716.435559] alg: No test for adler32 (adler32-zlib) [ 3717.397298] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 3717.558104] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 3719.191727] Key type lgssc registered [ 3720.412273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3728.579152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3728.621754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3730.171883] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3730.186603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3730.241658] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.19/24@tcp [ 3734.927396] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3739.638110] LustreError: 127267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3739.658646] LustreError: 127267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3744.096805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3744.475692] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3744.501158] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.19/24@tcp [ 3744.560894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3748.989057] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3756.460335] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3762.500785] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3770.284405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3770.738440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3774.861876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 3777.567693] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3784.962470] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3800.546801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3800.550540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3800.557122] LustreError: Skipped 1 previous similar message [ 3800.560191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3800.591331] Lustre: Skipped 1 previous similar message [ 3805.077768] Lustre: server umount lustre-OST0000 complete [ 3810.798027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.808708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3810.812148] Lustre: Skipped 1 previous similar message [ 3810.826846] Lustre: Skipped 2 previous similar messages [ 3815.841841] Lustre: server umount lustre-MDT0000 complete [ 3815.905382] LustreError: 127266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.243776] LustreError: 127249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788615976 with bad export cookie 10980592239458313723 [ 3820.244902] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3820.263469] LustreError: 127249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3820.705454] Lustre: server umount lustre-MDT0001 complete [ 3832.919934] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 3836.387472] Key type lgssc unregistered [ 3836.860443] LNet: 130852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3836.869691] LNetError: 130852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3836.894203] LNet: Removed LNI 192.168.203.119@tcp [ 3837.915715] Key type .llcrypt unregistered [ 3837.918226] Key type ._llcrypt unregistered [ 3864.767316] Key type ._llcrypt registered [ 3864.773135] Key type .llcrypt registered [ 3864.859053] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 3883.124854] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 3883.732043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3883.775964] alg: No test for adler32 (adler32-zlib) [ 3885.025598] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 3885.440335] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 3887.271146] Key type lgssc registered [ 3887.999751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3923.370882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3923.452313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3924.841785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3924.944092] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3925.102192] Lustre: lustre-MDT0000: new disk, initializing [ 3925.191770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3925.214971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3930.115429] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3939.470802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3939.574506] Lustre: 134251:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3939.627913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3939.633031] Lustre: Skipped 1 previous similar message [ 3939.735677] Lustre: lustre-MDT0001: new disk, initializing [ 3939.880770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3939.923853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3939.928152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3944.448605] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3955.789138] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3962.296914] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3969.126963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3969.501245] Lustre: lustre-OST0000: new disk, initializing [ 3969.507067] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3969.521967] Lustre: 136217:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3969.657331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3976.093758] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 3977.813319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3977.829967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3977.930283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3986.278282] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3992.125917] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3992.467768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3997.394895] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3997.644788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4003.814593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4003.816243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4003.839938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4003.865345] Lustre: Skipped 1 previous similar message [ 4006.348546] Lustre: server umount lustre-OST0000 complete [ 4011.488535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4011.499689] LustreError: Skipped 1 previous similar message [ 4011.510040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.528147] Lustre: Skipped 1 previous similar message [ 4011.538510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4014.048694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4014.053256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4014.110436] Lustre: Skipped 1 previous similar message [ 4018.754259] Lustre: server umount lustre-MDT0000 complete [ 4022.805951] LustreError: 134242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788616178 with bad export cookie 5064448192472227362 [ 4022.810522] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4022.821771] LustreError: 134242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4023.256126] Lustre: server umount lustre-MDT0001 complete [ 4033.181692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4034.008905] LustreError: 138437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4034.040413] LustreError: 138437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4034.176984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4039.653342] LustreError: 138437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4039.673752] LustreError: 138437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4039.735304] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4043.747575] LustreError: 138438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4048.876623] LustreError: 138437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4049.394312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4049.874626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4055.286320] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4063.567578] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4071.250205] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4079.851519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4080.292941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4088.486102] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4098.526849] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4104.760512] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4105.032876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4110.038802] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4110.236943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4120.761689] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4120.978798] Lustre: lustre-OST0001: new disk, initializing [ 4120.982993] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4120.989702] Lustre: 141811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4121.112626] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4128.079962] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4131.470046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4131.485340] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4131.637448] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4137.550610] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4137.875961] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4142.786908] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4143.031071] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4148.561862] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff920ad2afa800.ost_server_uuid 50 [ 4150.222376] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff920ad2afa800.ost_server_uuid in IDLE state after 0 sec [ 4161.484394] Lustre: server umount lustre-OST0001 complete [ 4162.530795] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4162.539437] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.552325] LustreError: 140840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4162.568876] LustreError: 140840:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4165.608304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4165.618179] LustreError: Skipped 1 previous similar message [ 4165.627211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4166.632577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4166.647673] Lustre: Skipped 2 previous similar messages [ 4171.695345] Lustre: server umount lustre-OST0000 complete [ 4177.889332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4177.891094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.900068] LustreError: Skipped 1 previous similar message [ 4177.913407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4177.930418] Lustre: Skipped 4 previous similar messages [ 4181.736141] Lustre: server umount lustre-MDT0000 complete [ 4183.009705] LustreError: 141821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4183.032279] LustreError: 141821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4185.206155] LustreError: 138417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788616340 with bad export cookie 5064448192472228244 [ 4185.209585] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4185.213498] LustreError: 138417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4185.629597] Lustre: server umount lustre-MDT0001 complete [ 4203.867568] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 4206.351723] Key type lgssc unregistered [ 4206.639310] LNet: 144813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4206.649667] LNetError: 144813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4206.669021] LNet: Removed LNI 192.168.203.119@tcp [ 4207.500854] Key type .llcrypt unregistered [ 4207.503078] Key type ._llcrypt unregistered [ 4248.470269] Key type ._llcrypt registered [ 4248.472946] Key type .llcrypt registered [ 4248.602961] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 4263.311677] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4264.722965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4264.740289] alg: No test for adler32 (adler32-zlib) [ 4265.796390] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 4266.064627] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 4267.831690] Key type lgssc registered [ 4268.928847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4301.959771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4301.983838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.220549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4303.264887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4303.378289] Lustre: lustre-MDT0000: new disk, initializing [ 4303.506705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4303.531206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4307.565264] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4317.595716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4317.724346] Lustre: 148960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4317.778379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4317.781910] Lustre: Skipped 1 previous similar message [ 4317.849671] Lustre: lustre-MDT0001: new disk, initializing [ 4317.981920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4318.064193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4318.082397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4323.122402] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4332.348335] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4337.287309] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4343.046699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4343.281809] Lustre: lustre-OST0000: new disk, initializing [ 4343.285520] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4343.292168] Lustre: 150930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4343.390617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4348.947991] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4351.549306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4351.562667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4351.696384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4357.616705] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4362.278233] Lustre: Failing over lustre-OST0000 [ 4362.393382] Lustre: server umount lustre-OST0000 complete [ 4379.154270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4379.404492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4379.421632] Lustre: Skipped 1 previous similar message [ 4379.444748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4379.462408] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4380.448155] Lustre: 147186:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788616520/real 1788616520] req@00000000cf3fd134 x1875500252299520/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788616536 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4380.822509] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4383.583243] Lustre: 147186:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788616523/real 1788616523] req@00000000500ef8c5 x1875500252300672/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788616539 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4384.333979] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4384.776996] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4384.777691] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 4387.956670] Lustre: Failing over lustre-MDT0000 [ 4388.125710] Lustre: server umount lustre-MDT0000 complete [ 4388.896158] Lustre: 147186:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788616528/real 1788616528] req@0000000097867d5c x1875500252301696/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788616544 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4388.938256] Lustre: 147186:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4389.858035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.869932] Lustre: Skipped 1 previous similar message [ 4394.079144] Lustre: 147186:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788616533/real 1788616533] req@00000000546bf73a x1875500252302720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788616549 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4394.985187] LustreError: 148973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4395.002989] LustreError: 148973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4396.147178] LustreError: 148969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.099044] LustreError: 148967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.111675] LustreError: 148967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4405.217846] LustreError: 148972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4405.244149] LustreError: 148972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4406.123558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4406.232583] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4406.239136] Lustre: 147185:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788616545/real 1788616545] req@0000000041656618 x1875500252310912/t0(0) o400->MGC192.168.203.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788616561 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4406.550225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4406.639267] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4411.001753] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4411.165224] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4411.887672] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4411.908240] Lustre: Skipped 1 previous similar message [ 4411.930268] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4411.973628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4420.582967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4420.594753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4420.616580] Lustre: Skipped 1 previous similar message [ 4420.626360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4422.119941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4422.137783] Lustre: Skipped 1 previous similar message [ 4425.837380] Lustre: server umount lustre-OST0000 complete [ 4432.354476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4432.367379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.374492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4432.389562] Lustre: Skipped 2 previous similar messages [ 4435.715412] Lustre: server umount lustre-MDT0000 complete [ 4437.473765] LustreError: 148968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4437.486567] LustreError: 148968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4439.490699] LustreError: 152184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788616595 with bad export cookie 12611340687671155466 [ 4439.492395] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4439.501801] LustreError: 152184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4439.740382] Lustre: server umount lustre-MDT0001 complete [ 4449.855852] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 4452.404498] Key type lgssc unregistered [ 4452.711800] LNet: 154581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4452.725900] LNetError: 154581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4452.751772] LNet: Removed LNI 192.168.203.119@tcp [ 4453.576295] Key type .llcrypt unregistered [ 4453.582362] Key type ._llcrypt unregistered [ 4489.242089] Key type ._llcrypt registered [ 4489.245228] Key type .llcrypt registered [ 4489.391955] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4490.540825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4490.562286] alg: No test for adler32 (adler32-zlib) [ 4491.565975] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 4491.789797] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 4493.447296] Key type lgssc registered [ 4494.497756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4502.114226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4502.140856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.565176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4508.182777] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4513.763838] LustreError: 156097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4513.780466] LustreError: 156097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4517.534831] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4518.882605] LustreError: 156098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4521.370589] LustreError: 156126:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4521.377506] LustreError: 156126:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4521.387588] LustreError: 156126:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4521.704765] Lustre: server umount lustre-MDT0000 complete [ 4538.318957] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 4548.057442] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4583.936877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.295598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4584.320560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4584.370729] Lustre: lustre-MDT0000: new disk, initializing [ 4584.461172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4584.476142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4590.306457] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4600.294025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4600.448458] Lustre: 159429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4600.546409] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4600.556254] Lustre: Skipped 1 previous similar message [ 4600.695461] Lustre: lustre-MDT0001: new disk, initializing [ 4600.841346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4600.911085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4600.922874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4605.209862] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4613.984812] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4619.141314] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4624.441018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4624.651257] Lustre: lustre-OST0000: new disk, initializing [ 4624.655893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4624.664348] Lustre: 161398:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4624.722151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4631.482871] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4634.191703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4634.206841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4634.323477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4640.414690] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4644.550374] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4646.055910] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:59:59 (1788616799) [ 4880.358118] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4880.361526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4880.388756] Lustre: Skipped 1 previous similar message [ 4880.410632] Lustre: Skipped 2 previous similar messages [ 4885.477434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4890.600092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4890.611686] Lustre: Skipped 4 previous similar messages [ 4893.151639] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4893.434830] Lustre: server umount lustre-MDT0000 complete [ 4895.712771] LustreError: 159436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4895.735910] LustreError: 159436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4897.947315] LustreError: 159422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788617053 with bad export cookie 5902458561068424010 [ 4897.947890] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4897.970213] LustreError: 159422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4900.831436] LustreError: 159437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4900.831742] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4900.860819] Lustre: Skipped 1 previous similar message [ 4900.864703] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4904.628966] Lustre: server umount lustre-MDT0001 complete [ 4914.574670] Lustre: server umount lustre-OST0000 complete [ 4922.890386] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 4934.615392] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 4970.037993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4970.277540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4970.306678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4970.385234] Lustre: lustre-MDT0000: new disk, initializing [ 4970.440715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4970.454648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4975.138282] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4985.384397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4985.470269] Lustre: 166069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4985.512615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4985.516422] Lustre: Skipped 1 previous similar message [ 4985.571346] Lustre: lustre-MDT0001: new disk, initializing [ 4985.653725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4985.678162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4985.690175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4990.271695] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 4998.501129] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5003.612772] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5010.036029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5010.305556] Lustre: lustre-OST0000: new disk, initializing [ 5010.309556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5010.316921] Lustre: 168036:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5010.442620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5016.152301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5016.168719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5016.278529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5017.303452] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5026.554396] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5031.107304] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5031.287907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5034.730734] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5034.909609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5037.027305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5037.028384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5037.037048] LustreError: Skipped 1 previous similar message [ 5037.038782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5037.038790] Lustre: Skipped 1 previous similar message [ 5037.074745] Lustre: Skipped 1 previous similar message [ 5047.271260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5047.299625] Lustre: Skipped 4 previous similar messages [ 5051.360720] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5051.603460] Lustre: server umount lustre-OST0000 complete [ 5057.515198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5057.531847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5067.746972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5067.756378] Lustre: Skipped 5 previous similar messages [ 5070.303167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5070.686204] Lustre: server umount lustre-MDT0000 complete [ 5072.865551] LustreError: 167106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5072.895818] LustreError: 167106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5075.034861] LustreError: 169579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788617230 with bad export cookie 5902458561068582840 [ 5075.038759] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5075.044955] LustreError: 169579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5075.564350] Lustre: server umount lustre-MDT0001 complete [ 5084.141329] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:07:17 (1788617237) [ 5100.960918] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 5109.236405] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5145.846254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5146.161452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5146.243906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5146.436866] Lustre: lustre-MDT0000: new disk, initializing [ 5146.591641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5146.612433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5151.604451] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5160.555417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5160.656764] Lustre: 172796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5160.847844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5165.543239] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5174.624483] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5180.207437] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5186.856582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5187.354637] Lustre: lustre-OST0000: new disk, initializing [ 5187.358519] Lustre: Skipped 1 previous similar message [ 5187.366436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5187.378174] Lustre: Skipped 2 previous similar messages [ 5187.388607] Lustre: 174775:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5194.009693] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5196.356316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5196.360053] Lustre: Skipped 1 previous similar message [ 5196.364371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5196.428623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5204.718847] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5217.258338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5217.266131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5217.289065] Lustre: Skipped 1 previous similar message [ 5217.293911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5217.302024] Lustre: Skipped 1 previous similar message [ 5221.487118] Lustre: server umount lustre-OST0000 complete [ 5227.489959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5227.503061] Lustre: Skipped 2 previous similar messages [ 5240.287174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5240.655578] Lustre: server umount lustre-MDT0000 complete [ 5242.848473] LustreError: 172806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5242.871476] LustreError: 172806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5244.292302] LustreError: 172786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788617400 with bad export cookie 5902458561068583722 [ 5244.298249] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5244.303030] LustreError: 172786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5244.734140] Lustre: server umount lustre-MDT0001 complete [ 5253.132643] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:10:06 (1788617406) [ 5269.801476] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 5280.067367] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5317.005985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5317.291642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5317.325712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5317.463242] Lustre: lustre-MDT0000: new disk, initializing [ 5317.558375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5317.566286] Lustre: Skipped 2 previous similar messages [ 5317.585391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5321.754852] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5332.036161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5332.131386] Lustre: 179098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5332.417025] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5336.873925] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5346.048426] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5352.459462] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5359.588935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5359.874593] Lustre: 181074:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5366.454846] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5367.894428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5368.018877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5376.399370] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5388.772202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5388.778195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5388.788298] LustreError: Skipped 2 previous similar messages [ 5388.803418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5388.803425] Lustre: Skipped 6 previous similar messages [ 5390.477163] Lustre: server umount lustre-OST0000 complete [ 5393.888687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5408.223118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5408.419395] Lustre: server umount lustre-MDT0000 complete [ 5409.250037] LustreError: 180266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5409.272874] LustreError: 180266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5412.363557] LustreError: 179089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788617568 with bad export cookie 5902458561068584807 [ 5412.369188] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5412.376149] LustreError: 179089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5412.752852] Lustre: server umount lustre-MDT0001 complete [ 5423.281328] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 5426.820917] Key type lgssc unregistered [ 5427.208851] LNet: 182931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5427.237922] LNetError: 182931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5428.260881] LNet: Removed LNI 192.168.203.119@tcp [ 5429.249378] Key type .llcrypt unregistered [ 5429.251697] Key type ._llcrypt unregistered [ 5452.171092] Key type ._llcrypt registered [ 5452.173574] Key type .llcrypt registered [ 5452.293804] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5453.192661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5453.216468] alg: No test for adler32 (adler32-zlib) [ 5454.322606] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 5454.625477] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 5456.295971] Key type lgssc registered [ 5457.358882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5464.798779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5464.833182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5466.289894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5470.601107] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5476.836418] LustreError: 184344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.851646] LustreError: 184344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5478.870568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5479.354451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5483.594830] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5490.976193] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5496.863977] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5503.558475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5503.996784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5510.330544] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5512.214820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5518.622402] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5539.812641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5539.819133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5539.850093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5542.884202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5542.885442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5542.886062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5544.293878] Lustre: server umount lustre-OST0000 complete [ 5551.072833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5551.086447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5551.104150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5551.116049] Lustre: Skipped 1 previous similar message [ 5554.772350] Lustre: server umount lustre-MDT0000 complete [ 5558.240327] LustreError: 184339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5558.270072] LustreError: 184339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5558.712847] LustreError: 184325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788617714 with bad export cookie 14190502345480840108 [ 5558.724897] LustreError: 184325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5558.729327] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5559.155867] Lustre: server umount lustre-MDT0001 complete [ 5571.020177] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 5574.234468] Key type lgssc unregistered [ 5574.590758] LNet: 187899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5574.601581] LNetError: 187899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5574.621456] LNet: Removed LNI 192.168.203.119@tcp [ 5575.709800] Key type .llcrypt unregistered [ 5575.716285] Key type ._llcrypt unregistered [ 5601.105785] Key type ._llcrypt registered [ 5601.107456] Key type .llcrypt registered [ 5601.196544] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 5602.462979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5602.701425] alg: No test for adler32 (adler32-zlib) [ 5603.831359] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 5604.148657] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 5605.815184] Key type lgssc registered [ 5607.067628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5614.689537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5614.703183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5616.053695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5620.719597] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5625.318658] LustreError: 189314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5625.341991] LustreError: 189314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5629.442974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5629.832827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5633.984590] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5641.198784] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5647.130867] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5654.446756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5654.866333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5661.046446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5661.148163] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5668.405953] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5679.588959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5679.589092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5679.607499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5679.622348] Lustre: Skipped 1 previous similar message [ 5679.643090] Lustre: Skipped 1 previous similar message [ 5680.283367] Lustre: lustre-OST0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 5683.484119] Lustre: server umount lustre-OST0000 complete [ 5690.828848] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff920ac5a10000.ost_server_uuid 50 [ 5692.332761] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5706.720039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5706.724857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5706.734543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5710.817806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5710.819728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.983636] Lustre: server umount lustre-MDT0000 complete [ 5715.492470] LustreError: 189294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788617871 with bad export cookie 13621776126840413671 [ 5715.515476] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5715.822868] Lustre: server umount lustre-MDT0001 complete [ 5724.261345] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:17:57 (1788617877) [ 5730.359846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5730.795124] LustreError: 193228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5730.859951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5734.879802] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5735.907604] LustreError: 193229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5741.027691] LustreError: 193228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.888231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5743.275548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5747.725796] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5755.030944] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5760.279102] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5766.904870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5767.229571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5773.366863] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5775.422212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5780.199710] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5786.635197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5786.861670] Lustre: lustre-OST0001: new disk, initializing [ 5786.874524] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5786.887820] Lustre: 196069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5786.950115] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5793.078937] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5794.425713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5794.442813] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5794.530366] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5802.644813] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5805.036204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5805.047595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5805.053205] Lustre: Skipped 1 previous similar message [ 5809.121937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5809.131242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5813.374518] Lustre: lustre-OST0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 5813.382090] Lustre: Skipped 4 previous similar messages [ 5819.359111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5819.485269] Lustre: server umount lustre-OST0000 complete [ 5820.395355] LustreError: 195048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5820.429170] LustreError: 195048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5825.942703] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5826.200106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5828.730933] LustreError: 195048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.754840] LustreError: 195048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5831.006718] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5831.297890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5845.984943] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5845.987444] LustreError: 195525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5845.992380] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5845.992854] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5845.992857] Lustre: Skipped 3 previous similar messages [ 5845.994369] LustreError: Skipped 1 previous similar message [ 5846.023017] LustreError: 195525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5851.721203] Lustre: server umount lustre-OST0001 complete [ 5855.713636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5861.747917] Lustre: server umount lustre-MDT0000 complete [ 5865.243686] LustreError: 193209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788618021 with bad export cookie 13621776126840414742 [ 5865.247230] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5865.255568] LustreError: 193209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5865.520854] Lustre: server umount lustre-MDT0001 complete [ 5889.314079] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:20:42 (1788618042) [ 5895.599171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5895.878601] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5895.916588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5896.120767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5900.631564] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5908.455554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5908.553745] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5908.590140] Lustre: 199507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5913.712154] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5920.650783] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5926.802484] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5933.135948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5933.347625] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5939.947389] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5940.372292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5947.534735] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5953.800755] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5953.878764] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5959.899616] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 5968.690822] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5973.986326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5974.012814] Lustre: Skipped 3 previous similar messages [ 5974.025154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5974.028101] Lustre: Skipped 8 previous similar messages [ 5978.419680] Lustre: server umount lustre-OST0000 complete [ 5979.112991] LustreError: 201359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5979.130381] LustreError: 201359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5994.474418] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5994.477658] LustreError: Skipped 1 previous similar message [ 6000.162631] Lustre: server umount lustre-OST0001 complete [ 6010.155869] Lustre: server umount lustre-MDT0000 complete [ 6013.936246] LustreError: 199499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788618169 with bad export cookie 13621776126840416149 [ 6013.943844] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6013.951214] LustreError: 199499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6014.203358] Lustre: server umount lustre-MDT0001 complete [ 6035.685241] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:23:09 (1788618189) [ 6051.804828] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 6060.744089] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 6092.787532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6092.957712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6093.000138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6093.104703] Lustre: lustre-MDT0000: new disk, initializing [ 6093.198659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6093.207182] Lustre: Skipped 3 previous similar messages [ 6093.219928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6097.778052] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6107.695654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6107.869386] Lustre: 207600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6107.928036] LustreError: 207610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6107.932216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6107.942779] LustreError: 207610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6107.955668] Lustre: Skipped 1 previous similar message [ 6108.073792] Lustre: lustre-MDT0001: new disk, initializing [ 6108.231441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6108.256104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6112.593788] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6121.266127] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6127.019535] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6134.346765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6134.797881] Lustre: lustre-OST0000: new disk, initializing [ 6134.809215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6134.820202] Lustre: 209571:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6141.657692] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6143.596817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6143.611666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6143.742881] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6151.166237] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6155.980859] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6156.212402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6160.989408] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6161.181740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6164.452102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6164.456078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6164.477185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6164.486231] Lustre: Skipped 6 previous similar messages [ 6164.492966] Lustre: Skipped 11 previous similar messages [ 6169.358345] Lustre: server umount lustre-OST0000 complete [ 6179.495543] Lustre: server umount lustre-MDT0000 complete [ 6183.530996] LustreError: 207591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788618339 with bad export cookie 13621776126840417710 [ 6183.533385] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6183.543354] LustreError: 207591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6183.881772] Lustre: server umount lustre-MDT0001 complete [ 6192.746940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6197.777994] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6205.493377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6210.099441] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6217.965734] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6223.495065] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6229.817968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6230.141276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6230.147073] Lustre: Skipped 4 previous similar messages [ 6236.725902] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6243.516534] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6247.360375] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6247.534457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6251.457451] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6251.688794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6255.084843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6255.085694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6255.101204] Lustre: Skipped 3 previous similar messages [ 6255.116558] LustreError: Skipped 2 previous similar messages [ 6259.601281] Lustre: server umount lustre-OST0000 complete [ 6265.110194] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6269.387913] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6297.254907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6303.321275] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6310.128575] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6317.948678] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 6319.977669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.987173] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.988020] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6320.010287] Lustre: Skipped 1 previous similar message [ 6323.306581] Lustre: server umount lustre-OST0000 complete [ 6328.806727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6328.817071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6328.819165] Lustre: Skipped 4 previous similar messages [ 6337.582915] Lustre: server umount lustre-MDT0000 complete [ 6340.577506] Lustre: 188625:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788618480/real 1788618480] req@0000000095a787c9 x1875501655513216/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788618496 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6340.583921] LustreError: 212515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6340.617955] Lustre: 188625:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6340.653177] LustreError: 212515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6341.232172] LustreError: 211768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788618496 with bad export cookie 13621776126840418585 [ 6341.239121] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6341.244753] LustreError: 211768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6341.701250] Lustre: server umount lustre-MDT0001 complete [ 6349.765584] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:28:23 (1788618503) [ 6355.307817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6359.291875] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6367.482646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6372.942328] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6380.916470] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6387.222414] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6394.613189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6402.131157] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6410.165129] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6415.281321] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6415.639055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6420.587613] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6420.830944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6427.830486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6428.090719] Lustre: lustre-OST0001: new disk, initializing [ 6428.094202] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6428.098189] Lustre: 221124:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6434.444855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6434.456881] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6434.562338] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6434.877664] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6443.824360] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6448.877921] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6449.078875] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6454.205749] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6454.517547] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6460.388287] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6460.394610] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6460.398557] LustreError: Skipped 1 previous similar message [ 6460.419518] Lustre: Skipped 4 previous similar messages [ 6462.741586] Lustre: server umount lustre-OST0001 complete [ 6469.243286] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6469.444873] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6474.396254] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6474.598520] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6502.043262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6502.246874] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6502.254341] Lustre: Skipped 5 previous similar messages [ 6507.405586] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6507.511611] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6507.519715] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6507.526489] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6514.448209] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6515.811437] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6523.618748] Lustre: server umount lustre-OST0001 complete [ 6546.670547] Lustre: server umount lustre-OST0000 complete [ 6556.552970] Lustre: server umount lustre-MDT0000 complete [ 6560.302272] LustreError: 217734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788618716 with bad export cookie 13621776126840419558 [ 6560.304182] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6560.313771] LustreError: 217734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6566.667745] Lustre: server umount lustre-MDT0001 complete [ 6590.327301] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:32:23 (1788618743) [ 6597.157402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6597.464128] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6597.511893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6601.984549] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6610.315492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6610.432235] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6610.469374] Lustre: 226369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6615.027502] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6621.186700] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6626.035672] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6633.021329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6633.170593] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6639.307832] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6647.111912] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6655.673037] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6655.777947] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6661.122687] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6668.251734] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6670.421710] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6675.040865] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6675.232666] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6679.747996] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6680.002333] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6686.527911] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff920ac8020000.ost_server_uuid 50 [ 6688.247764] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff920ac8020000.ost_server_uuid in IDLE state after 0 sec [ 6689.426729] Lustre: Permanently deactivating lustre-OST0001 [ 6689.464596] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6693.568200] Lustre: Permanently reactivating lustre-OST0001 [ 6693.594973] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6693.610697] Lustre: Skipped 2 previous similar messages [ 6698.982644] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6699.001720] Lustre: Skipped 18 previous similar messages [ 6703.525946] Lustre: server umount lustre-OST0001 complete [ 6704.630630] LustreError: 228218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6704.662836] LustreError: 228218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 6713.308846] Lustre: server umount lustre-OST0000 complete [ 6718.432355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6718.448206] LustreError: Skipped 8 previous similar messages [ 6718.455142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6718.477321] Lustre: Skipped 10 previous similar messages [ 6723.150659] Lustre: server umount lustre-MDT0000 complete [ 6726.668864] LustreError: 226362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788618882 with bad export cookie 13621776126840420839 [ 6726.672224] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6726.681359] LustreError: 226362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6727.271629] Lustre: server umount lustre-MDT0001 complete [ 6752.910920] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:35:06 (1788618906) [ 6763.088718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6763.348901] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6763.371326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6763.381151] Lustre: Skipped 2 previous similar messages [ 6770.142092] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6779.528258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6779.666933] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6779.719757] Lustre: 233064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6785.321579] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6792.588625] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6799.134700] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6806.692442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6806.949563] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6806.997607] Lustre: Permanently deactivating lustre-OST0000 [ 6807.032200] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6813.418304] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6817.315988] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6821.262649] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6828.591828] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6828.687582] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6836.253440] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6838.031250] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6845.136328] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6847.545277] Lustre: Permanently reactivating lustre-OST0000 [ 6847.572139] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6847.581266] Lustre: Skipped 2 previous similar messages [ 6853.226570] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6853.226879] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 0000000067f6bdd1, cur 1788619008 deadline 1788619003 last 1788618973 [ 6853.237604] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6853.277970] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6853.289705] Lustre: Skipped 1 previous similar message [ 6853.369609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6870.400254] Lustre: server umount lustre-OST0001 complete [ 6881.054115] Lustre: server umount lustre-OST0000 complete [ 6891.379535] Lustre: server umount lustre-MDT0000 complete [ 6894.803576] LustreError: 233057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788619050 with bad export cookie 13621776126840422484 [ 6894.812491] LustreError: 233057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6894.813620] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6895.230128] Lustre: server umount lustre-MDT0001 complete [ 6906.540712] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 6909.697873] Key type lgssc unregistered [ 6909.997716] LNet: 237946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6910.013274] LNetError: 237946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6911.080967] LNet: Removed LNI 192.168.203.119@tcp [ 6912.147305] Key type .llcrypt unregistered [ 6912.150648] Key type ._llcrypt unregistered [ 6935.850161] Key type ._llcrypt registered [ 6935.854424] Key type .llcrypt registered [ 6935.969397] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 6937.133408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6937.490761] alg: No test for adler32 (adler32-zlib) [ 6938.743206] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 6939.250703] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 6940.968732] Key type lgssc registered [ 6942.467340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6953.162441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6953.177411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6954.538460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6959.471531] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6964.712340] LustreError: 239415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6964.727224] LustreError: 239415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6968.767802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6968.966226] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6968.995989] Lustre: 239404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6969.012433] Lustre: Permanently deactivating lustre-MDT0001 [ 6969.034205] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6969.320658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6974.395836] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 6976.026083] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6982.533294] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6988.922857] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6996.553406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6997.067944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7000.115423] LustreError: 241257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7000.135096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7004.088347] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7005.164135] LustreError: 241428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7010.280511] LustreError: 241428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7012.789420] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7013.377719] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7018.481803] LustreError: 241258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7018.508748] LustreError: 241258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7019.693047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7019.930509] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7025.151897] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7025.167389] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7027.634295] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7036.175921] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7039.685988] Lustre: Permanently reactivating lustre-MDT0001 [ 7039.704661] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7039.710240] Lustre: Skipped 2 previous similar messages [ 7042.708611] Lustre: lustre-MDT0001: Client 8de55fd9-d1f6-41d7-8bd0-27be274abe3d (at 192.168.203.19@tcp) reconnecting [ 7049.417257] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7049.938825] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7049.950063] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7049.962779] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7050.006626] LustreError: 243178:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7050.010085] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7050.026217] LustreError: 243178:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7050.026352] LustreError: 243178:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.119@tcp: cfg command failed: rc = -108 [ 7050.027400] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7050.027400] [ 7050.029644] LustreError: 239407:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7051.163668] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7057.981099] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7058.223759] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7062.760442] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7063.079497] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7066.826427] Lustre: Permanently deactivating lustre-MDT0001 [ 7066.851630] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7066.855902] Lustre: Skipped 2 previous similar messages [ 7072.982053] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7081.441600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7081.446933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7081.459401] Lustre: Skipped 2 previous similar messages [ 7081.500318] Lustre: Skipped 3 previous similar messages [ 7083.509981] Lustre: server umount lustre-MDT0000 complete [ 7086.568465] LustreError: 240421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7086.586393] LustreError: 240421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7087.146832] LustreError: 239395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788619242 with bad export cookie 1753706663077676202 [ 7087.169946] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7087.171129] LustreError: 239395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7087.584599] Lustre: server umount lustre-MDT0001 complete [ 7093.305902] Lustre: server umount lustre-OST0000 complete [ 7106.084041] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:40:59 (1788619259) [ 7117.492469] Lustre: server umount lustre-OST0001 complete [ 7124.441131] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 7135.610720] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7173.298585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7173.556196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7173.567247] Lustre: Skipped 2 previous similar messages [ 7173.600362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7173.695504] Lustre: lustre-MDT0000: new disk, initializing [ 7173.800955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7173.815603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7178.498587] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7189.269837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7189.433056] Lustre: 247672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7189.522719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7189.525878] Lustre: Skipped 1 previous similar message [ 7189.633229] Lustre: lustre-MDT0001: new disk, initializing [ 7189.766172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7189.816142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7189.828487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7195.041554] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7205.531868] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7211.976516] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7219.673460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7220.101246] Lustre: lustre-OST0000: new disk, initializing [ 7220.106195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7220.114634] Lustre: 249645:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7220.236620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7225.428954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7225.443416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7225.649760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7227.946283] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7238.000392] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7240.275064] LustreError: 249236:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7249.774439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7249.981780] Lustre: lustre-OST0001: new disk, initializing [ 7249.986774] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7249.995713] Lustre: 250758:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7250.319070] LustreError: 249236:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7250.363604] LustreError: 247680:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7255.652732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7255.671042] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7255.801361] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7256.561185] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7260.417913] LustreError: 247680:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7265.139823] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7266.651450] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7271.408907] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7271.410137] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7271.411952] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7271.420072] LustreError: Skipped 1 previous similar message [ 7271.430876] Lustre: Skipped 2 previous similar messages [ 7274.790123] Lustre: server umount lustre-OST0001 complete [ 7276.528705] LustreError: 249628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7276.547931] LustreError: 249628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7292.385990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7292.388517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7292.395700] LustreError: Skipped 1 previous similar message [ 7292.397622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7292.412767] Lustre: Skipped 1 previous similar message [ 7292.414367] Lustre: Skipped 1 previous similar message [ 7297.011196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7297.023376] Lustre: Skipped 2 previous similar messages [ 7298.515582] Lustre: server umount lustre-OST0000 complete [ 7302.625357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7302.635876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7302.669329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7309.010421] Lustre: server umount lustre-MDT0000 complete [ 7310.311843] LustreError: 250900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7310.320896] LustreError: 250900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7312.957084] LustreError: 247665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788619468 with bad export cookie 1753706663077679016 [ 7312.971684] LustreError: 247665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7312.973144] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7319.414892] Lustre: server umount lustre-MDT0001 complete [ 7330.887294] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 7333.895142] Key type lgssc unregistered [ 7334.259406] LNet: 252798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7334.266729] LNetError: 252798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7334.301763] LNet: Removed LNI 192.168.203.119@tcp [ 7335.327980] Key type .llcrypt unregistered [ 7335.330646] Key type ._llcrypt unregistered [ 7376.760622] Key type ._llcrypt registered [ 7376.764275] Key type .llcrypt registered [ 7376.881169] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7377.954307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7378.009045] alg: No test for adler32 (adler32-zlib) [ 7379.386683] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 7379.648063] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 7381.327787] Key type lgssc registered [ 7382.628916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7390.766507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7390.802839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7392.087250] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7392.122254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7392.393213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7396.996387] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7406.787768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7406.924234] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7406.964244] Lustre: 255000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7407.209173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7412.085777] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7420.806777] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7427.047764] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7433.708129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7433.900472] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7434.037958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7440.183622] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7442.020259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7448.286504] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7503.331586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7503.338427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7503.342975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7503.352163] Lustre: Skipped 1 previous similar message [ 7503.364760] Lustre: Skipped 1 previous similar message [ 7505.636874] Lustre: server umount lustre-OST0000 complete [ 7509.964677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7522.642352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7522.943722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7523.975715] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7524.003904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7524.005680] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7524.042152] Lustre: Skipped 1 previous similar message [ 7529.807797] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7536.510691] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7580.650130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7580.650904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7580.654706] LustreError: Skipped 1 previous similar message [ 7580.666617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7580.675170] Lustre: Skipped 1 previous similar message [ 7580.715755] Lustre: Skipped 1 previous similar message [ 7582.643318] Lustre: server umount lustre-OST0000 complete [ 7590.883622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7590.890223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7590.903687] Lustre: Skipped 1 previous similar message [ 7596.000845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7596.032300] Lustre: Skipped 2 previous similar messages [ 7601.120385] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7601.121496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7601.153213] Lustre: Skipped 1 previous similar message [ 7601.461276] Lustre: server umount lustre-MDT0000 complete [ 7605.345199] LustreError: 254992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788619761 with bad export cookie 17455214756577857569 [ 7605.348239] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7605.366136] LustreError: 254992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7605.784264] Lustre: server umount lustre-MDT0001 complete [ 7617.682402] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 7620.885298] Key type lgssc unregistered [ 7621.131181] LNet: 260427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7621.139357] LNetError: 260427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7621.158714] LNet: Removed LNI 192.168.203.119@tcp [ 7622.040891] Key type .llcrypt unregistered [ 7622.048103] Key type ._llcrypt unregistered [ 7644.904308] Key type ._llcrypt registered [ 7644.906125] Key type .llcrypt registered [ 7645.004788] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7646.144200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7646.237400] alg: No test for adler32 (adler32-zlib) [ 7647.358906] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 7647.647282] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 7649.343640] Key type lgssc registered [ 7650.508870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7657.855572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7657.874366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.221805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7662.841787] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7668.709391] LustreError: 261841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7668.738236] LustreError: 261841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7672.026885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7672.647137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7677.966803] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7687.711333] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7693.097190] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7699.387074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7699.922514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7703.990597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7706.083830] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7713.274691] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7742.944388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7742.948096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.950902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7742.978211] Lustre: Skipped 1 previous similar message [ 7747.820115] Lustre: server umount lustre-OST0000 complete [ 7753.186082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7753.190881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7753.207467] Lustre: Skipped 1 previous similar message [ 7753.230814] Lustre: Skipped 2 previous similar messages [ 7757.031688] Lustre: server umount lustre-MDT0000 complete [ 7758.312335] LustreError: 261840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7758.330530] LustreError: 261840:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7760.927173] LustreError: 261822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788619916 with bad export cookie 166244834410336813 [ 7760.935328] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7760.941803] LustreError: 261822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7761.309914] Lustre: server umount lustre-MDT0001 complete [ 7771.227817] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 7773.843536] Key type lgssc unregistered [ 7774.181371] LNet: 266146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7774.189646] LNetError: 266146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7775.205810] LNet: Removed LNI 192.168.203.119@tcp [ 7776.121783] Key type .llcrypt unregistered [ 7776.123400] Key type ._llcrypt unregistered [ 7791.019235] Key type ._llcrypt registered [ 7791.022801] Key type .llcrypt registered [ 7791.129802] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7792.017990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7792.124663] alg: No test for adler32 (adler32-zlib) [ 7793.189486] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 7793.466477] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 7795.159841] Key type lgssc registered [ 7796.126509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7802.668036] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7802.696835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7804.272868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7808.830799] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7813.601414] LustreError: 267344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7813.629510] LustreError: 267344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7816.677177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7817.135972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7821.262725] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7827.880279] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7833.336871] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7839.937850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7840.355165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7845.990769] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7848.581361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7855.214911] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7867.360429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7867.363490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7867.400784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7869.414203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7869.418967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7869.436679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7869.468298] Lustre: Skipped 1 previous similar message [ 7872.812820] Lustre: server umount lustre-OST0000 complete [ 7878.626772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7878.637646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7878.664123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7884.768238] LustreError: 268632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7884.808243] LustreError: 268632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7885.366034] Lustre: server umount lustre-MDT0000 complete [ 7889.030676] LustreError: 267324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620044 with bad export cookie 8343429917092611248 [ 7889.041344] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7889.044782] LustreError: 267324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7889.341403] Lustre: server umount lustre-MDT0001 complete [ 7898.874536] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 7901.348053] Key type lgssc unregistered [ 7901.624150] LNet: 271023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7901.634309] LNetError: 271023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7901.646832] LNet: Removed LNI 192.168.203.119@tcp [ 7902.470242] Key type .llcrypt unregistered [ 7902.474076] Key type ._llcrypt unregistered [ 7925.322309] Key type ._llcrypt registered [ 7925.325781] Key type .llcrypt registered [ 7925.452734] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 7926.465948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7926.563322] alg: No test for adler32 (adler32-zlib) [ 7927.708317] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 7927.928879] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 7929.583274] Key type lgssc registered [ 7930.716761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7938.379698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7938.406906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7939.863550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7943.784246] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7949.285150] LustreError: 272436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7949.304288] LustreError: 272436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7952.670545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7952.942586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7957.118954] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7964.257083] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7970.087532] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7976.256315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7976.715537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7981.733562] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 7984.970200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7988.337798] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8013.793895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8013.800150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8013.823104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8015.840153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8015.846594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8015.850062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8019.421227] Lustre: server umount lustre-OST0000 complete [ 8024.544615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8024.551805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8024.578808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8024.581462] Lustre: Skipped 1 previous similar message [ 8029.414958] Lustre: server umount lustre-MDT0000 complete [ 8031.201657] LustreError: 272431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8031.223120] LustreError: 272431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8033.138828] LustreError: 272416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620188 with bad export cookie 13297886893671365072 [ 8033.144222] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8033.153857] LustreError: 272416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8033.673352] Lustre: server umount lustre-MDT0001 complete [ 8044.878719] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 8048.138455] Key type lgssc unregistered [ 8048.509666] LNet: 276741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8048.523221] LNetError: 276741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8048.540702] LNet: Removed LNI 192.168.203.119@tcp [ 8049.452830] Key type .llcrypt unregistered [ 8049.453987] Key type ._llcrypt unregistered [ 8069.372338] Key type ._llcrypt registered [ 8069.374702] Key type .llcrypt registered [ 8069.500215] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8070.903341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8070.935421] alg: No test for adler32 (adler32-zlib) [ 8071.979872] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 8072.194504] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 8073.871211] Key type lgssc registered [ 8075.150058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8083.057725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8083.107660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8084.943716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8089.745705] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8095.212275] LustreError: 277974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8095.251907] LustreError: 277974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8099.471168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8100.069873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8103.989812] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8110.167180] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8115.804154] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8122.472062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8122.772754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8128.388771] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8129.987063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8136.180646] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8150.496857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8150.504704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8150.509565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8150.535471] LustreError: Skipped 1 previous similar message [ 8150.550653] Lustre: Skipped 1 previous similar message [ 8153.470232] Lustre: server umount lustre-OST0000 complete [ 8160.736619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8160.753560] Lustre: Skipped 1 previous similar message [ 8160.769280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8160.776187] Lustre: Skipped 1 previous similar message [ 8163.747523] Lustre: server umount lustre-MDT0000 complete [ 8165.859810] LustreError: 277956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8165.888918] LustreError: 277956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8168.167337] LustreError: 277941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620323 with bad export cookie 865533756840437866 [ 8168.179526] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8168.188547] LustreError: 277941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8168.635275] Lustre: server umount lustre-MDT0001 complete [ 8179.013470] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 8181.863507] Key type lgssc unregistered [ 8182.193894] LNet: 281675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8182.203704] LNetError: 281675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8182.240558] LNet: Removed LNI 192.168.203.119@tcp [ 8183.350416] Key type .llcrypt unregistered [ 8183.356147] Key type ._llcrypt unregistered [ 8196.335503] Key type ._llcrypt registered [ 8196.337753] Key type .llcrypt registered [ 8196.503371] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8214.607041] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 8230.020691] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8231.422418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8231.471136] alg: No test for adler32 (adler32-zlib) [ 8232.554413] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 8232.799337] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 8234.472054] Key type lgssc registered [ 8235.628873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8271.639656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8271.659217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8273.008222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8273.081908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8273.281743] Lustre: lustre-MDT0000: new disk, initializing [ 8273.494679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8273.532300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8278.981896] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8288.333258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8288.452976] Lustre: 285225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8288.526380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8288.528852] Lustre: Skipped 1 previous similar message [ 8288.619482] Lustre: lustre-MDT0001: new disk, initializing [ 8288.746239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8288.813738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8288.828434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8293.118298] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8301.604142] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8307.560379] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8314.232394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8314.502288] Lustre: lustre-OST0000: new disk, initializing [ 8314.513076] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8314.519518] Lustre: 287205:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8314.600530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8321.314519] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8322.114105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8322.121119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8322.296905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8330.505987] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8335.457636] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8335.660951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8340.017115] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8340.189184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8343.010493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8343.011489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8343.022694] LustreError: Skipped 1 previous similar message [ 8343.045159] Lustre: Skipped 1 previous similar message [ 8343.052518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8348.132261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8348.143028] Lustre: Skipped 1 previous similar message [ 8348.311506] Lustre: server umount lustre-OST0000 complete [ 8353.251173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.257725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8353.265717] Lustre: Skipped 1 previous similar message [ 8353.283982] Lustre: Skipped 2 previous similar messages [ 8358.270796] Lustre: server umount lustre-MDT0000 complete [ 8361.603728] LustreError: 285218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620517 with bad export cookie 17394330407663135133 [ 8361.610912] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8361.614919] LustreError: 285218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8361.910839] Lustre: server umount lustre-MDT0001 complete [ 8369.219600] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:02:02 (1788620522) [ 8375.527187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8375.975459] LustreError: 289615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8375.993133] LustreError: 289615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8376.049644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8381.158411] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8381.410712] LustreError: 289616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8381.448844] LustreError: 289616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8385.510603] LustreError: 289615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8389.396664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8389.911954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8394.346871] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8401.494492] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8407.271713] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8414.730501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8415.075408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8421.468682] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8429.116308] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8444.897256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8444.898782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8444.906209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8444.916610] Lustre: Skipped 1 previous similar message [ 8444.928267] Lustre: Skipped 1 previous similar message [ 8448.860889] Lustre: server umount lustre-OST0000 complete [ 8455.141684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8455.166395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8455.173830] Lustre: Skipped 1 previous similar message [ 8458.839343] Lustre: server umount lustre-MDT0000 complete [ 8460.263917] LustreError: 289615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8460.284620] LustreError: 289615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8462.142529] LustreError: 289596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620617 with bad export cookie 17394330407663136015 [ 8462.160088] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8462.504593] Lustre: server umount lustre-MDT0001 complete [ 8473.145695] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 8475.762582] Key type lgssc unregistered [ 8476.061528] LNet: 293174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8476.073247] LNetError: 293174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8477.115686] LNet: Removed LNI 192.168.203.119@tcp [ 8478.139505] Key type .llcrypt unregistered [ 8478.150679] Key type ._llcrypt unregistered [ 8503.312235] Key type ._llcrypt registered [ 8503.314925] Key type .llcrypt registered [ 8503.428345] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 8517.977359] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8518.985640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8519.117063] alg: No test for adler32 (adler32-zlib) [ 8520.149688] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 8520.398350] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 8522.071173] Key type lgssc registered [ 8523.109994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8564.730807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8564.772595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8566.008584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8566.045539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8566.154213] Lustre: lustre-MDT0000: new disk, initializing [ 8566.207966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8566.222873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8570.986418] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8581.634890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8581.844302] Lustre: 296997:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8581.935366] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8581.942475] Lustre: Skipped 1 previous similar message [ 8582.080232] Lustre: lustre-MDT0001: new disk, initializing [ 8582.212671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8582.274294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8582.286446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8586.942770] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8595.712465] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8601.071708] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8607.816305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8608.123132] Lustre: lustre-OST2710: new disk, initializing [ 8608.129729] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8608.141720] Lustre: 298964:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8608.251217] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8614.881458] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8617.019483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8617.029487] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8617.121645] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8624.071195] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8631.289178] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8631.453652] Lustre: lustre-OST03e8: new disk, initializing [ 8631.462227] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8631.467728] Lustre: 300029:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8631.586771] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8638.302769] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8641.203944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8641.223050] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8641.287117] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8646.294767] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8653.358620] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8653.566307] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8659.261725] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8659.544984] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8664.899966] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8665.230224] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8670.555689] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8670.762261] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8676.468646] Lustre: server umount lustre-MDT0000 complete [ 8677.352835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8677.368919] Lustre: Skipped 1 previous similar message [ 8680.454568] LustreError: 296988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620836 with bad export cookie 13047381937144006281 [ 8680.463890] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8680.472104] LustreError: 296988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8680.916128] Lustre: server umount lustre-MDT0001 complete [ 8694.501071] Lustre: server umount lustre-OST2710 complete [ 8698.464606] Lustre: 294792:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788620838/real 1788620838] req@00000000fac7a59f x1875504714025728/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788620854 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8698.498106] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8698.510518] Lustre: Skipped 2 previous similar messages [ 8699.069097] Lustre: server umount lustre-OST03e8 complete [ 8721.306402] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 8731.657518] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8769.590310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8769.864404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8769.906250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8770.007781] Lustre: lustre-MDT0000: new disk, initializing [ 8770.129389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8770.145539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8776.327342] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8789.533791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8789.702851] Lustre: 305275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8789.762531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8789.766871] Lustre: Skipped 1 previous similar message [ 8789.879356] Lustre: lustre-MDT0001: new disk, initializing [ 8789.974752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8790.006812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8790.017268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8795.466349] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8805.563040] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8812.292635] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8819.670877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8819.907895] Lustre: lustre-OST0000: new disk, initializing [ 8819.913702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8819.920395] Lustre: 307243:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8820.009315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8825.929360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8825.956788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8826.167846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8827.264659] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8837.890430] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8843.227479] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8843.560231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8849.018250] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8849.261203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8851.941214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8851.945826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8851.950631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8851.953141] LustreError: Skipped 1 previous similar message [ 8851.993535] Lustre: Skipped 1 previous similar message [ 8857.077688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8857.094419] Lustre: Skipped 2 previous similar messages [ 8857.708472] Lustre: server umount lustre-OST0000 complete [ 8861.664217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8861.673282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8861.696090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8867.301144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8867.318688] Lustre: Skipped 3 previous similar messages [ 8867.893849] Lustre: server umount lustre-MDT0000 complete [ 8871.753794] LustreError: 305267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621027 with bad export cookie 13047381937144007961 [ 8871.756344] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8871.770115] LustreError: 305267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8872.241680] Lustre: server umount lustre-MDT0001 complete [ 8881.610159] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:10:34 (1788621034) [ 8883.149981] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8885.066976] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:10:38 (1788621038) [ 8889.352557] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 8918.806528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8919.108166] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8919.145034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8919.384474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8924.492544] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8933.745029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8933.893592] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8933.922336] Lustre: 311103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8939.521100] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 8962.104520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8962.397920] Lustre: Denying initial registration attempt from nid 192.168.203.119@tcp, specified as failover [ 8962.410761] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8962.415912] LustreError: 312556:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 8962.424603] LustreError: 312556:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8962.440481] LustreError: 312556:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8962.525553] Lustre: server umount lustre-OST0000 complete [ 8962.529875] LustreError: 312556:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8970.208590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8970.221585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8970.253883] Lustre: Skipped 1 previous similar message [ 8970.264726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8975.870524] Lustre: server umount lustre-MDT0000 complete [ 8978.931631] LustreError: 311113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8978.947358] LustreError: 311113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8979.955620] LustreError: 312926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621135 with bad export cookie 13047381937144008850 [ 8979.962538] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8979.968287] LustreError: 312926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8980.381596] Lustre: server umount lustre-MDT0001 complete [ 8991.811955] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 8994.591534] Key type lgssc unregistered [ 8995.088947] LNet: 313676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8995.095126] LNetError: 313676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8995.107888] LNet: Removed LNI 192.168.203.119@tcp [ 8995.841956] Key type .llcrypt unregistered [ 8995.847419] Key type ._llcrypt unregistered [ 9008.144539] Key type ._llcrypt registered [ 9008.147221] Key type .llcrypt registered [ 9008.267467] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 9009.414414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9009.537700] alg: No test for adler32 (adler32-zlib) [ 9010.708820] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 9011.007763] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 9012.695156] Key type lgssc registered [ 9013.830416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9061.397962] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 9072.501526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9072.523733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9074.141509] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9074.228036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9074.592139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9081.126541] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9090.436153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9090.619943] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9090.674652] Lustre: 316445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9090.808566] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9090.972650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9097.017357] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9104.559212] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9109.734563] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9128.497338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9128.739914] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9128.950903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9134.948428] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9140.986325] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9143.279127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9143.280819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9143.286731] Lustre: Skipped 1 previous similar message [ 9143.298401] Lustre: Skipped 1 previous similar message [ 9145.056945] Lustre: server umount lustre-OST0000 complete [ 9148.385326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9148.404547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9148.411697] Lustre: Skipped 1 previous similar message [ 9153.505381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9153.510377] Lustre: Skipped 1 previous similar message [ 9154.826866] Lustre: server umount lustre-MDT0000 complete [ 9159.326986] LustreError: 316438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621315 with bad export cookie 15073686686734071032 [ 9159.334076] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9159.344634] LustreError: 316438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9159.757549] Lustre: server umount lustre-MDT0001 complete [ 9170.371835] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 9173.217736] Key type lgssc unregistered [ 9173.511686] LNet: 320019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9173.519629] LNetError: 320019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9173.542848] LNet: Removed LNI 192.168.203.119@tcp [ 9174.718140] Key type .llcrypt unregistered [ 9174.720857] Key type ._llcrypt unregistered [ 9200.712801] Key type ._llcrypt registered [ 9200.716881] Key type .llcrypt registered [ 9200.851313] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 9202.207630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9202.255827] alg: No test for adler32 (adler32-zlib) [ 9203.427995] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 9203.684485] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 9205.368504] Key type lgssc registered [ 9206.579163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9213.898924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9213.931399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9215.500975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9220.030804] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9225.698133] LustreError: 321434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9225.713464] LustreError: 321434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9228.285530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9228.708501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9233.049238] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9240.330106] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9245.879942] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9254.768380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9255.025219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9261.279354] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9268.241312] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9284.065931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9284.076220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9284.096599] Lustre: Skipped 1 previous similar message [ 9286.317069] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.19@tcp (stopping) [ 9286.327743] Lustre: Skipped 2 previous similar messages [ 9288.471837] Lustre: server umount lustre-MDT0000 complete [ 9289.190571] LustreError: 322161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9291.384748] LustreError: 321429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9291.403163] LustreError: 321429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9291.955115] LustreError: 321415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621447 with bad export cookie 18325786351900775003 [ 9291.962480] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9291.979567] LustreError: 321415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9294.304667] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9294.312338] LustreError: 321430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9294.325805] Lustre: Skipped 1 previous similar message [ 9294.327061] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9296.354232] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9298.626301] Lustre: server umount lustre-MDT0001 complete [ 9305.137746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9313.560552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9316.839962] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfe524597c6bf6e93 [ 9316.852422] Lustre: MGC192.168.203.119@tcp: Connection restored to 0@lo (at 0@lo) [ 9317.046042] LustreError: 325045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9317.064108] LustreError: 325045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9317.320123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9317.411933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9322.384841] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9326.564872] LustreError: 325046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9326.584188] LustreError: 325046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9326.592736] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9330.904790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9331.353415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9336.066329] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9336.819725] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9343.562937] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9345.181605] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9351.891888] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9353.525879] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9358.304394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9358.311200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9358.322945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9358.327396] Lustre: Skipped 1 previous similar message [ 9361.890700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9361.910050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9364.212896] Lustre: server umount lustre-OST0000 complete [ 9372.128717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9372.140718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9372.164742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9372.167690] Lustre: Skipped 3 previous similar messages [ 9374.569831] Lustre: server umount lustre-MDT0000 complete [ 9377.790581] LustreError: 325050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9377.831062] LustreError: 325050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9378.946494] LustreError: 321415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621534 with bad export cookie 18325786351900782227 [ 9378.947666] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9378.969224] LustreError: 321415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9379.218466] Lustre: server umount lustre-MDT0001 complete [ 9392.175985] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 9394.877081] Key type lgssc unregistered [ 9395.138951] LNet: 327719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9395.160576] LNetError: 327719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9395.193238] LNet: Removed LNI 192.168.203.119@tcp [ 9396.254619] Key type .llcrypt unregistered [ 9396.256599] Key type ._llcrypt unregistered [ 9422.415104] Key type ._llcrypt registered [ 9422.419680] Key type .llcrypt registered [ 9422.548445] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 9423.546708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9423.651735] alg: No test for adler32 (adler32-zlib) [ 9424.871904] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 9425.061533] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 9426.783290] Key type lgssc registered [ 9428.287791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9436.249601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9436.279548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9437.783213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9442.534397] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9447.907062] LustreError: 329133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9447.938205] LustreError: 329133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9451.174763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9451.600953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9456.463114] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9464.408399] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9470.443386] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9476.258732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9476.296960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9476.302219] Lustre: Skipped 1 previous similar message [ 9479.629365] Lustre: server umount lustre-MDT0000 complete [ 9481.696983] LustreError: 329134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9481.741328] LustreError: 329134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9483.526512] LustreError: 329868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621639 with bad export cookie 17876412285120243798 [ 9483.537190] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9483.549720] LustreError: 329868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9483.870110] Lustre: server umount lustre-MDT0001 complete [ 9498.505601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9498.867691] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9498.903691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9499.167128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9503.445139] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9511.617038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9511.801901] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9511.867730] Lustre: 331719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9512.276574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9517.855784] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9526.426850] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9532.187205] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9542.587389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9542.856657] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9542.880192] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9542.892329] LustreError: 333605:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 9542.899203] LustreError: 333605:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9542.904268] LustreError: 333605:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9542.962423] Lustre: server umount lustre-OST0000 complete [ 9542.969876] LustreError: 333605:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9552.693455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9552.863770] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9553.018388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9559.045557] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9561.821717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9568.479659] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9577.702198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9577.825251] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9577.966649] Lustre: lustre-OST0001: new disk, initializing [ 9577.976052] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9577.984876] Lustre: 335146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9578.089842] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9583.711785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9583.723786] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9583.840491] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9586.109306] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9594.235347] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9599.466556] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9599.467355] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9599.479122] Lustre: Skipped 1 previous similar message [ 9599.480298] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9599.509443] Lustre: Skipped 1 previous similar message [ 9603.292989] Lustre: server umount lustre-OST0001 complete [ 9604.579983] LustreError: 334071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9604.611038] LustreError: 334071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9607.649341] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9607.662077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9607.675049] Lustre: Skipped 1 previous similar message [ 9607.686637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9609.702738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9609.704209] LustreError: 334073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9609.718758] Lustre: Skipped 2 previous similar messages [ 9609.752563] LustreError: 334073:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9612.945643] Lustre: server umount lustre-OST0000 complete [ 9619.950562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9619.952232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9619.960852] LustreError: Skipped 1 previous similar message [ 9619.967249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9619.970607] Lustre: Skipped 2 previous similar messages [ 9619.992432] Lustre: Skipped 1 previous similar message [ 9622.903218] Lustre: server umount lustre-MDT0000 complete [ 9625.058578] LustreError: 331732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9625.071365] LustreError: 331732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9626.793962] LustreError: 331713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621782 with bad export cookie 17876412285120244344 [ 9626.800746] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9626.810688] LustreError: 331713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9627.155440] Lustre: server umount lustre-MDT0001 complete [ 9637.893241] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [ 9641.211846] Key type lgssc unregistered [ 9641.619556] LNet: 337103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9641.627666] LNetError: 337103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9642.663946] LNet: Removed LNI 192.168.203.119@tcp [ 9643.752145] Key type .llcrypt unregistered [ 9643.755622] Key type ._llcrypt unregistered [ 9714.292154] Key type ._llcrypt registered [ 9714.293869] Key type .llcrypt registered [ 9714.424886] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [ 9728.683540] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [ 9729.547743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9729.562631] alg: No test for adler32 (adler32-zlib) [ 9730.504706] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 9730.690063] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [ 9732.359164] Key type lgssc registered [ 9733.516726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9773.911323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9773.967457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9775.328042] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9775.372448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9775.495111] Lustre: lustre-MDT0000: new disk, initializing [ 9775.591709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9775.618409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9781.071973] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9792.811355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9793.029143] Lustre: 342037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9793.100752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9793.105558] Lustre: Skipped 1 previous similar message [ 9793.199297] Lustre: lustre-MDT0001: new disk, initializing [ 9793.281809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9793.322708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9793.328816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9797.857774] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9807.930444] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9814.381501] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9821.370085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9821.632340] Lustre: lustre-OST0000: new disk, initializing [ 9821.636493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9821.644502] Lustre: 344005:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9821.753783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9828.383255] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9831.489591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9831.526593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9831.612523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9837.284565] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9842.037970] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9842.258565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9847.053507] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9847.294815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9852.396945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9852.408138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9852.439485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9852.442253] Lustre: Skipped 1 previous similar message [ 9855.538038] Lustre: server umount lustre-OST0000 complete [ 9860.065324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9860.077987] LustreError: Skipped 1 previous similar message [ 9860.086883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9860.099112] Lustre: Skipped 1 previous similar message [ 9860.108068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9862.626941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9862.629248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9862.654593] Lustre: Skipped 1 previous similar message [ 9865.792926] Lustre: server umount lustre-MDT0000 complete [ 9870.051060] LustreError: 344004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622025 with bad export cookie 17737781770588383080 [ 9870.052353] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9870.069108] LustreError: 344004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9870.513701] Lustre: server umount lustre-MDT0001 complete [ 9879.825911] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:27:13 (1788622033) [ 9888.337323] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:27:21 (1788622041) [ 9897.039154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9897.638159] LustreError: 346762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9897.671394] LustreError: 346762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9897.741349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9903.044301] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9903.077358] LustreError: 346763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9908.193101] LustreError: 346762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9912.227108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9912.644961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9917.788989] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9924.764273] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9930.486115] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9938.311754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9938.849792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9945.519320] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [ 9953.113873] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9964.981355] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9966.884379] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9968.409891] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9970.178625] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9971.972829] Lustre: Failing over lustre-MDT0000 [ 9972.328754] Lustre: server umount lustre-MDT0000 complete [ 9972.707336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9972.710687] LustreError: 346759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9972.716021] Lustre: Skipped 2 previous similar messages [ 9972.744764] LustreError: 346759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9977.825438] LustreError: 347754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9977.845824] LustreError: 347754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9986.678986] LustreError: 346759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9986.717173] LustreError: 346759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9989.088279] Lustre: 340261:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788622128/real 1788622128] req@0000000039cdd918 x1875505982803968/t0(0) o400->MGC192.168.203.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788622144 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9989.122124] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9991.700709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9998.601793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9998.655491] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9999.513099] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10002.712945] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10003.953469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10004.028517] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10004.092987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10011.969641] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10013.443501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10017.095381] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10018.760797] Lustre: Failing over lustre-MDT0000 [10019.201275] Lustre: server umount lustre-MDT0000 complete [10019.298021] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10019.302914] LustreError: 346759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10019.318221] Lustre: Skipped 2 previous similar messages [10019.348921] LustreError: 346759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [10035.681495] Lustre: 340260:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788622175/real 1788622175] req@00000000db94a0f4 x1875505982823552/t0(0) o400->MGC192.168.203.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788622191 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10035.711934] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10038.363647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10045.933505] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf629428aa351f1fd [10045.946705] LustreError: 351183:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.119@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000008fc35361) refcount nonzero (1) after lock cleanup; forcing cleanup. [10045.968316] Lustre: MGC192.168.203.119@tcp: Connection restored to 0@lo (at 0@lo) [10045.981352] Lustre: Skipped 2 previous similar messages [10046.068209] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.19@tcp (not set up) [10046.297349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10046.343886] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10051.190378] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10051.354614] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10051.588479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10051.644709] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10051.743175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10061.864449] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10063.624447] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10072.037201] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10072.042295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10072.048566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10072.068407] Lustre: Skipped 1 previous similar message [10076.499492] Lustre: server umount lustre-MDT0000 complete [10077.153955] LustreError: 346781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10077.180928] LustreError: 346781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [10080.110505] LustreError: 346743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622235 with bad export cookie 17737781770588385789 [10080.122424] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10080.133592] LustreError: 346743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10080.518037] Lustre: server umount lustre-MDT0001 complete [10094.759733] Lustre: server umount lustre-OST0000 complete [10110.087356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10110.687420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10115.427978] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10123.990583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10129.041887] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10136.587289] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10141.930081] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10148.912032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10154.697536] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10155.323165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10161.876855] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10167.757720] Lustre: server umount lustre-OST0000 complete [10175.457169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10175.476194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10175.499126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10175.503804] Lustre: Skipped 2 previous similar messages [10186.210944] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10186.403536] Lustre: server umount lustre-MDT0000 complete [10187.231546] Lustre: 340258:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788622326/real 1788622326] req@0000000049d1b13b x1875505982874496/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788622342 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10187.277692] LustreError: 353853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10187.299367] LustreError: 353853:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10189.931276] LustreError: 353107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622345 with bad export cookie 17737781770588386545 [10189.939217] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10189.946441] LustreError: 353107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10190.356281] Lustre: server umount lustre-MDT0001 complete [10200.563319] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [10203.270934] Key type lgssc unregistered [10203.700470] LNet: 356658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10203.710522] LNetError: 356658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10204.774918] LNet: Removed LNI 192.168.203.119@tcp [10205.594235] Key type .llcrypt unregistered [10205.597889] Key type ._llcrypt unregistered [10231.678539] Key type ._llcrypt registered [10231.680881] Key type .llcrypt registered [10231.800594] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [10233.049864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10233.258954] alg: No test for adler32 (adler32-zlib) [10234.458940] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [10234.679581] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [10236.339324] Key type lgssc registered [10237.802730] Lustre: Echo OBD driver; http://www.lustre.org/ [10246.039485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10246.058079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10247.540651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10252.042927] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10257.898743] LustreError: 358169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10257.949906] LustreError: 358169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10261.233495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10261.694206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10266.739807] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10274.297388] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10280.452841] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10287.432982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10287.969119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10293.115869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10294.614778] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10302.378743] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10318.305538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10318.314836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10318.332114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10321.892821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10321.893124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10321.905969] Lustre: Skipped 1 previous similar message [10321.918197] Lustre: Skipped 2 previous similar messages [10327.012244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10327.018781] Lustre: Skipped 2 previous similar messages [10332.127135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10332.132231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10332.156876] Lustre: Skipped 2 previous similar messages [10332.341972] Lustre: server umount lustre-MDT0000 complete [10335.927605] LustreError: 358149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622491 with bad export cookie 6598534035358402023 [10335.938074] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10336.164612] Lustre: server umount lustre-MDT0001 complete [10350.376760] Lustre: server umount lustre-OST0000 complete [10364.760886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10365.298651] LustreError: 361903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10365.420549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10370.489701] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10370.536649] LustreError: 361904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10375.662322] LustreError: 361903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10379.593262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10380.071363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10384.313818] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10391.396231] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10396.754737] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10402.807995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10403.160187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10409.482822] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10410.370475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10417.348439] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10430.945993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10430.948965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10430.971250] Lustre: Skipped 2 previous similar messages [10430.980026] Lustre: Skipped 2 previous similar messages [10435.427027] Lustre: server umount lustre-MDT0000 complete [10436.068733] LustreError: 361903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10436.096569] LustreError: 361903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10439.126325] LustreError: 361883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622594 with bad export cookie 6598534035358403136 [10439.137413] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10439.149910] LustreError: 361883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10439.679531] Lustre: server umount lustre-MDT0001 complete [10454.106332] Lustre: server umount lustre-OST0000 complete [10471.228097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10471.761703] LustreError: 365818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10471.896819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10476.923935] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10485.979164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10491.803779] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10499.938465] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10506.130461] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10513.505885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10513.969542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10513.977491] Lustre: Skipped 1 previous similar message [10517.060634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10520.875681] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10528.798555] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10536.416950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10536.438856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10536.455682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10536.468590] Lustre: Skipped 1 previous similar message [10536.482155] Lustre: Skipped 1 previous similar message [10540.236514] Lustre: server umount lustre-OST0000 complete [10546.656777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10546.679502] Lustre: Skipped 1 previous similar message [10550.618830] Lustre: server umount lustre-MDT0000 complete [10551.791989] LustreError: 365818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10551.827541] LustreError: 365818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10554.152205] LustreError: 365799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622709 with bad export cookie 6598534035358404158 [10554.154647] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10554.178076] LustreError: 365799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10554.804788] Lustre: server umount lustre-MDT0001 complete [10567.288604] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [10570.768142] Key type lgssc unregistered [10571.168077] LNet: 369349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10571.189552] LNetError: 369349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10571.221084] LNet: Removed LNI 192.168.203.119@tcp [10572.625526] Key type .llcrypt unregistered [10572.630660] Key type ._llcrypt unregistered [10600.063592] Key type ._llcrypt registered [10600.065523] Key type .llcrypt registered [10600.193338] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [10601.355261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10601.404242] alg: No test for adler32 (adler32-zlib) [10602.468056] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [10602.721584] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [10604.423098] Key type lgssc registered [10606.108395] Lustre: Echo OBD driver; http://www.lustre.org/ [10613.638973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10613.665650] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10613.676260] LustreError: 370792:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10613.705957] LustreError: 370792:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10613.714850] LustreError: 370792:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10613.723761] LustreError: 370792:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10613.733211] LustreError: 370792:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10620.714610] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10620.727727] LustreError: 371189:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10620.748339] LustreError: 371189:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10620.760307] LustreError: 371189:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10620.770459] LustreError: 371189:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10620.783333] LustreError: 371189:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10640.003401] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [10642.706995] Key type lgssc unregistered [10642.982784] LNet: 372192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10642.995900] LNetError: 372192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10643.015008] LNet: Removed LNI 192.168.203.119@tcp [10643.821562] Key type .llcrypt unregistered [10643.824174] Key type ._llcrypt unregistered [10662.738532] Key type ._llcrypt registered [10662.740142] Key type .llcrypt registered [10662.844525] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [10678.695126] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [10679.734911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10679.834674] alg: No test for adler32 (adler32-zlib) [10681.006737] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [10681.333388] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [10683.083415] Key type lgssc registered [10684.595983] Lustre: Echo OBD driver; http://www.lustre.org/ [10720.946809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10720.962893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10722.310146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10722.362051] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10722.459474] Lustre: lustre-MDT0000: new disk, initializing [10722.598887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10722.652193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10727.711963] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10738.435028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10738.608963] Lustre: 375344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10738.692763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10738.697954] Lustre: Skipped 1 previous similar message [10738.849819] Lustre: lustre-MDT0001: new disk, initializing [10739.004722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10739.090147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10739.123306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10743.340096] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10751.618729] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10756.886960] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10763.736450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10764.065229] Lustre: lustre-OST0000: new disk, initializing [10764.076061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10764.090947] Lustre: 377312:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10764.171396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10769.998097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10770.023749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10770.209922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10770.211954] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10779.185711] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10783.914106] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10784.211870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10788.788870] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10789.052942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10796.008370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10796.010918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10796.020138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10796.036933] Lustre: Skipped 1 previous similar message [10796.053914] Lustre: Skipped 1 previous similar message [10799.017138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10799.031867] Lustre: Skipped 1 previous similar message [10804.196605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10804.210144] Lustre: Skipped 1 previous similar message [10805.215155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10805.433518] Lustre: server umount lustre-OST0000 complete [10809.326747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10809.334492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10809.350089] Lustre: Skipped 1 previous similar message [10809.361153] Lustre: Skipped 1 previous similar message [10814.433417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10814.448422] Lustre: Skipped 1 previous similar message [10815.057994] Lustre: server umount lustre-MDT0000 complete [10818.389047] LustreError: 375337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622974 with bad export cookie 12936127199000181307 [10818.392500] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10818.397181] LustreError: 375337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10818.656842] Lustre: server umount lustre-MDT0001 complete [10826.090221] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10827.818764] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:43:01 (1788622981) [10834.124632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10834.500962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10839.412379] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10839.523288] LustreError: 379774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10839.540279] LustreError: 379774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10844.660608] LustreError: 379773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10848.470336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10848.904446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10853.471978] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10859.932918] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10864.877676] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10872.411418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10872.829218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10878.126793] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10887.759582] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10894.502619] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10894.691672] Lustre: lustre-OST0001: new disk, initializing [10894.696541] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10894.705654] Lustre: 382611:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10894.775059] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10899.865450] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [10901.105210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10901.151896] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10901.355893] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10908.702803] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10915.491351] Lustre: lustre-OST0001: Not available for connect from 192.168.203.19@tcp (stopping) [10916.843310] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10916.847488] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10916.860689] LustreError: Skipped 2 previous similar messages [10918.308782] Lustre: server umount lustre-OST0001 complete [10920.558888] LustreError: 381596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.093849] LustreError: 381596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.111881] LustreError: 381596:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10932.198844] LustreError: 381596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10932.229078] LustreError: 381596:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10935.270912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10935.279987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10935.291355] Lustre: Skipped 1 previous similar message [10935.303139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10935.315463] Lustre: Skipped 2 previous similar messages [10940.606629] Lustre: server umount lustre-OST0000 complete [10946.016426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10946.026891] LustreError: Skipped 1 previous similar message [10946.032304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10946.051189] Lustre: Skipped 1 previous similar message [10958.303127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10958.504551] Lustre: server umount lustre-MDT0000 complete [10961.759107] LustreError: 379753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623117 with bad export cookie 12936127199000182182 [10961.763728] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10961.772699] LustreError: 379753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10976.223212] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10976.513323] Lustre: server umount lustre-MDT0001 complete [10985.540710] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [10987.759898] Key type lgssc unregistered [10988.016390] LNet: 384600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10988.041233] LNetError: 384600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10989.102170] LNet: Removed LNI 192.168.203.119@tcp [10990.036793] Key type .llcrypt unregistered [10990.039273] Key type ._llcrypt unregistered [11041.620724] Key type ._llcrypt registered [11041.622981] Key type .llcrypt registered [11041.744545] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [11042.733121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11042.773622] alg: No test for adler32 (adler32-zlib) [11044.017231] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [11044.318485] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [11045.999434] Key type lgssc registered [11047.205525] Lustre: Echo OBD driver; http://www.lustre.org/ [11055.120736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11055.185138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11056.500783] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11056.525820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11056.716545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11061.223348] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11069.773715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11069.904039] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11069.971726] Lustre: 387394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11070.219845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11075.010975] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11081.444905] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11086.559957] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11090.912685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11090.924994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11090.952065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11091.428380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11091.464200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11091.476327] Lustre: Skipped 1 previous similar message [11096.700976] Lustre: server umount lustre-MDT0000 complete [11100.326581] LustreError: 387388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623256 with bad export cookie 6004584124765280557 [11100.329836] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11100.340991] LustreError: 387388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11100.752394] Lustre: server umount lustre-MDT0001 complete [11110.202419] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:47:43 (1788623263) [11116.967489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11117.529913] LustreError: 389907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11117.539921] LustreError: 389907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11117.581814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11122.360366] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11122.657137] LustreError: 389908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11127.781703] LustreError: 389907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11130.992867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11131.339976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11135.831045] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11143.557536] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11149.452161] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11156.353082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11156.541357] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11156.717339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11162.577959] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11169.780033] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11177.837224] Lustre: Permanently deactivating lustre-OST0000 [11177.848899] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11185.183970] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11187.306359] LustreError: 392894:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11187.322142] LustreError: 392894:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11187.329482] LustreError: 392894:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11190.221261] LustreError: 392942:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11190.240525] LustreError: 392942:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11198.056527] Lustre: server umount lustre-OST0000 complete [11200.622985] LustreError: 393192:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11200.627688] LustreError: 393192:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11203.042231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11203.055549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11203.084172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11206.627417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11206.629086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11211.745987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11211.759537] Lustre: Skipped 2 previous similar messages [11212.845107] Lustre: server umount lustre-MDT0000 complete [11216.574706] LustreError: 389887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623372 with bad export cookie 6004584124765281131 [11216.580019] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11216.589726] LustreError: 389887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11227.271124] Lustre: server umount lustre-MDT0001 complete [11235.724837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11240.384400] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11243.022742] LustreError: 394354:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11250.298453] Lustre: 394693:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.119@tcp [11252.668464] Lustre: server umount MGS complete [11264.517594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11264.906148] LustreError: 395417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11264.922810] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11264.927076] Lustre: Skipped 1 previous similar message [11264.999210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11269.332457] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11270.113474] LustreError: 395418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11277.149092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11277.461281] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11281.932924] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11288.599697] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11294.421736] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11296.268311] Lustre: Permanently reactivating lustre-OST0000 [11296.286618] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11296.295146] Lustre: Skipped 2 previous similar messages [11325.407767] LustreError: 397246:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11332.198478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11332.563478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11332.567600] Lustre: Skipped 1 previous similar message [11338.002165] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11344.555977] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11349.985723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11349.996132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11350.014354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11355.783330] Lustre: server umount lustre-OST0000 complete [11359.713546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11359.721360] LustreError: Skipped 1 previous similar message [11359.730938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11359.755324] Lustre: Skipped 1 previous similar message [11365.586463] LustreError: 399122:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000008d59b57a) refcount nonzero (2) after lock cleanup; forcing cleanup. [11365.784095] Lustre: server umount lustre-MDT0000 complete [11366.369670] LustreError: 396145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11366.394704] LustreError: 396145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11369.513117] LustreError: 395398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623525 with bad export cookie 6004584124765282188 [11369.524256] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11369.532520] LustreError: 395398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11370.092555] Lustre: server umount lustre-MDT0001 complete [11382.718563] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [11385.933896] Key type lgssc unregistered [11386.527700] LNet: 399873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11386.538075] LNetError: 399873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11386.576288] LNet: Removed LNI 192.168.203.119@tcp [11387.554375] Key type .llcrypt unregistered [11387.559829] Key type ._llcrypt unregistered [11407.807811] Key type ._llcrypt registered [11407.810279] Key type .llcrypt registered [11407.922076] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [11422.701890] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [11424.200179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11424.249053] alg: No test for adler32 (adler32-zlib) [11425.459855] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [11425.851969] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [11427.575591] Key type lgssc registered [11428.546822] Lustre: Echo OBD driver; http://www.lustre.org/ [11464.818933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11464.845781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11466.205784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11466.277737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11466.350217] Lustre: lustre-MDT0000: new disk, initializing [11466.441514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11466.469492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11471.228537] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11480.604148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11480.733549] Lustre: 403028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11480.811954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11480.816386] Lustre: Skipped 1 previous similar message [11480.877586] Lustre: lustre-MDT0001: new disk, initializing [11480.975090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11481.004613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11481.019929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11485.466736] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11493.801857] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11498.932381] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11505.133139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11505.392660] Lustre: lustre-OST0000: new disk, initializing [11505.401302] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11505.411961] Lustre: 404993:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11505.521634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11510.963368] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11514.423471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11514.437036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11514.550228] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11518.861800] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11522.740220] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11522.943991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11527.176444] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11527.378589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11530.211266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11530.211824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11530.228340] LustreError: Skipped 1 previous similar message [11530.229554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11530.244892] Lustre: Skipped 1 previous similar message [11535.260322] Lustre: server umount lustre-OST0000 complete [11540.448393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11540.463118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11540.467853] Lustre: Skipped 2 previous similar messages [11545.000036] Lustre: server umount lustre-MDT0000 complete [11548.226072] LustreError: 403018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623703 with bad export cookie 1227465991771164160 [11548.242285] LustreError: 403018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11548.243170] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11548.548390] Lustre: server umount lustre-MDT0001 complete [11556.885419] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11559.145405] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:55:11 (1788623711) [11566.728431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11567.228595] LustreError: 407455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11567.246460] LustreError: 407455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11567.297816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11572.717605] LustreError: 407456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11573.140701] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11577.828415] LustreError: 407455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11580.705071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11581.089487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11585.455828] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11591.853277] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11597.493804] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11603.611509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11603.903925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11610.532321] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11618.107262] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11619.918679] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11622.182780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11628.516323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11628.526059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11628.539355] Lustre: Skipped 2 previous similar messages [11628.539987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11633.692045] Lustre: server umount lustre-OST0000 complete [11638.756197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11638.773482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11638.776587] Lustre: Skipped 2 previous similar messages [11643.708674] Lustre: server umount lustre-MDT0000 complete [11643.872727] LustreError: 407451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11643.884661] LustreError: 407451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11646.385441] LustreError: 407436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623802 with bad export cookie 1227465991771165042 [11646.390104] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11646.395104] LustreError: 407436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11646.624592] Lustre: server umount lustre-MDT0001 complete [11655.639224] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [11657.926939] Key type lgssc unregistered [11658.176150] LNet: 411179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11658.184986] LNetError: 411179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11659.241395] LNet: Removed LNI 192.168.203.119@tcp [11660.110655] Key type .llcrypt unregistered [11660.114944] Key type ._llcrypt unregistered [11677.906817] Key type ._llcrypt registered [11677.909567] Key type .llcrypt registered [11677.995516] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [11678.263482] Key type .llcrypt unregistered [11678.265878] Key type ._llcrypt unregistered [11695.848853] Key type ._llcrypt registered [11695.852626] Key type .llcrypt registered [11695.957422] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [11696.757455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11696.826436] alg: No test for adler32 (adler32-zlib) [11697.898662] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [11698.173104] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [11699.831169] Key type lgssc registered [11701.076431] Lustre: Echo OBD driver; http://www.lustre.org/ [11709.515970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11709.538302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11711.095990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11715.449289] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11720.683975] LustreError: 413384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11720.705709] LustreError: 413384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11723.889934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11724.180093] LustreError: 413385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11724.200397] LustreError: 413385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11724.206142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11729.379471] LustreError: 413379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11729.390391] LustreError: 413379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11730.240934] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11734.498655] LustreError: 413379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11734.525862] LustreError: 413379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11735.552715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11737.104975] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11740.651397] LustreError: 413385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11740.668930] LustreError: 413385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11743.432322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11743.993392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11747.590551] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11756.002910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11756.010664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11756.035120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11757.999838] Lustre: server umount lustre-OST0000 complete [11763.173987] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11763.192826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11763.214795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11764.122535] Lustre: server umount lustre-MDT0000 complete [11764.711586] LustreError: 413385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11764.715246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11767.920258] LustreError: 413367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623923 with bad export cookie 5421925606533101575 [11767.924118] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11767.929485] LustreError: 413367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11777.668445] Lustre: server umount lustre-MDT0001 complete [11787.868820] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [11790.653798] Key type lgssc unregistered [11790.898841] LNet: 416527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11790.905966] LNetError: 416527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11790.924433] LNet: Removed LNI 192.168.203.119@tcp [11791.639714] Key type .llcrypt unregistered [11791.642886] Key type ._llcrypt unregistered [11813.888159] Key type ._llcrypt registered [11813.890324] Key type .llcrypt registered [11814.013389] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [11815.243754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11815.309315] alg: No test for adler32 (adler32-zlib) [11816.422132] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [11816.664562] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [11818.391253] Key type lgssc registered [11819.722776] Lustre: Echo OBD driver; http://www.lustre.org/ [11828.242757] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11828.278608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11843.871431] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11848.993541] LustreError: 417916:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.119@tcp: failed processing log, type 4: rc = -110 [11875.615258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11881.945819] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11889.620714] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11896.482463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11897.078648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11897.175370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11901.337087] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11902.441535] LustreError: 417942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11902.458854] LustreError: 417942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11907.556523] LustreError: 417941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11909.943245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11910.398840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11914.952583] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [11925.145504] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11932.465312] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11941.357982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11941.366678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11941.394340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11942.335310] Lustre: server umount lustre-OST0000 complete [11946.465867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11946.473134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11946.485318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11947.489304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11947.493501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11952.609974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11952.620047] Lustre: Skipped 2 previous similar messages [11955.796443] Lustre: server umount lustre-MDT0000 complete [11957.730714] LustreError: 418952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11957.743897] LustreError: 418952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11959.175962] LustreError: 417922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624114 with bad export cookie 4795040116584971498 [11959.183217] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11959.188942] LustreError: 417922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11959.523303] Lustre: server umount lustre-MDT0001 complete [11968.676986] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [11971.523734] Key type lgssc unregistered [11971.861727] LNet: 421494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11971.871377] LNetError: 421494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11972.901748] LNet: Removed LNI 192.168.203.119@tcp [11973.793675] Key type .llcrypt unregistered [11973.797693] Key type ._llcrypt unregistered [11997.577961] Key type ._llcrypt registered [11997.580765] Key type .llcrypt registered [11997.700833] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [11998.730955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11998.764119] alg: No test for adler32 (adler32-zlib) [11999.847084] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [12000.113678] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [12001.791490] Key type lgssc registered [12003.200299] Lustre: Echo OBD driver; http://www.lustre.org/ [12010.506906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12010.551094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12011.940745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12016.587572] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12022.254451] LustreError: 422910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12022.293534] LustreError: 422910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12024.029395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12024.466366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12029.193448] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12037.545285] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12042.907403] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12051.108557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12051.359104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12057.207709] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12057.517480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12064.316212] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12067.812040] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12067.813554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12067.830941] Lustre: Skipped 2 previous similar messages [12067.836610] Lustre: Skipped 1 previous similar message [12069.926810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12069.929607] Lustre: Skipped 3 previous similar messages [12071.064198] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.19@tcp (stopping) [12073.991448] Lustre: server umount lustre-MDT0000 complete [12074.980206] LustreError: 422909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12074.999610] LustreError: 422909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12076.160971] LustreError: 422891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.19@tcp arrived at 1788624231 with bad export cookie 16431840074224319731 [12076.170389] LustreError: 422891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12077.727227] LustreError: 422890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624233 with bad export cookie 16431840074224318842 [12077.736532] LustreError: 422890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12077.736566] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12080.096625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12080.098252] LustreError: 422905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12080.131793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12080.158669] LustreError: 422905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12082.152142] LustreError: 422909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12082.176549] LustreError: 422909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12083.959309] Lustre: server umount lustre-OST0000 complete [12087.269703] LustreError: 422905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12087.282312] LustreError: 422905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12090.463370] LustreError: 423639:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000dd2907c x1875508361985920/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12090.710145] Lustre: server umount lustre-MDT0001 complete [12101.899579] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12104.711634] Key type lgssc unregistered [12104.945447] LNet: 426592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12104.956952] LNetError: 426592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12104.977828] LNet: Removed LNI 192.168.203.119@tcp [12105.763930] Key type .llcrypt unregistered [12105.767110] Key type ._llcrypt unregistered [12129.055348] Key type ._llcrypt registered [12129.057787] Key type .llcrypt registered [12129.276786] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [12130.730242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12130.826725] alg: No test for adler32 (adler32-zlib) [12132.041926] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [12132.488091] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [12134.267497] Key type lgssc registered [12135.642903] Lustre: Echo OBD driver; http://www.lustre.org/ [12146.154051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12146.191271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12147.907991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12153.144649] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12157.413230] LustreError: 428003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12157.437913] LustreError: 428003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12160.360337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12160.723950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12164.137811] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12170.808752] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12176.175452] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12183.220135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12183.493273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12189.520845] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12192.750441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12196.235469] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12201.956637] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12201.975430] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12201.997161] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12202.997511] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12203.027684] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12206.056353] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12206.075488] Lustre: Skipped 3 previous similar messages [12206.287370] Lustre: server umount lustre-MDT0001 complete [12208.265880] LustreError: 427999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12211.178739] LustreError: 428004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12211.211157] LustreError: 428004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12216.295272] LustreError: 428003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12216.314640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12216.360459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12219.741788] Lustre: server umount lustre-OST0000 complete [12221.409253] LustreError: 428004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12224.694769] Lustre: server umount lustre-MDT0000 complete [12238.322411] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12241.272872] Key type lgssc unregistered [12241.538525] LNet: 431684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12241.550877] LNetError: 431684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12242.602610] LNet: Removed LNI 192.168.203.119@tcp [12243.420866] Key type .llcrypt unregistered [12243.425370] Key type ._llcrypt unregistered [12267.669737] Key type ._llcrypt registered [12267.672402] Key type .llcrypt registered [12267.777079] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [12282.715972] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [12283.691553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12283.846442] alg: No test for adler32 (adler32-zlib) [12284.867071] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [12285.187934] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [12286.887624] Key type lgssc registered [12287.917434] Lustre: Echo OBD driver; http://www.lustre.org/ [12323.889929] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [12338.833996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12338.894729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12340.155177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12340.181790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12340.314832] Lustre: lustre-MDT0000: new disk, initializing [12340.402622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12340.425253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12344.039650] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12356.546885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12356.806839] Lustre: lustre-OST0000: new disk, initializing [12356.815747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12356.827434] Lustre: Skipped 1 previous similar message [12356.837153] Lustre: 436718:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12356.900409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12362.834967] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12366.958360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12366.970321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12366.996966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12371.608788] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12387.156659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12387.292875] Lustre: 435701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12387.403667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12387.490954] Lustre: lustre-MDT0001: new disk, initializing [12387.629825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12387.673660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12387.687984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12392.442268] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12401.739400] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12401.955569] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12406.948657] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12407.229072] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12413.420969] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12413.421053] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12413.438701] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12413.472163] Lustre: Skipped 1 previous similar message [12415.624113] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.19@tcp (stopping) [12418.355734] Lustre: server umount lustre-MDT0001 complete [12423.653472] LustreError: 435707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12423.687399] LustreError: 435707:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12428.770390] LustreError: 437430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12428.806415] LustreError: 437430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12433.890102] LustreError: 435709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12433.922169] LustreError: 435709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12438.499954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12438.505925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12438.524476] Lustre: Skipped 1 previous similar message [12438.535715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12439.013917] LustreError: 435712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12442.630859] Lustre: server umount lustre-OST0000 complete [12444.131352] LustreError: 435713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12452.898637] Lustre: server umount lustre-MDT0000 complete [12469.110635] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12472.451573] Key type lgssc unregistered [12472.691741] LNet: 440466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12472.699442] LNetError: 440466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12473.785357] LNet: Removed LNI 192.168.203.119@tcp [12474.653218] Key type .llcrypt unregistered [12474.657322] Key type ._llcrypt unregistered [12513.981385] Key type ._llcrypt registered [12513.984861] Key type .llcrypt registered [12514.150582] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [12514.401506] Key type .llcrypt unregistered [12514.404361] Key type ._llcrypt unregistered [12531.015103] Key type ._llcrypt registered [12531.018162] Key type .llcrypt registered [12531.144323] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [12532.189956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12532.322902] alg: No test for adler32 (adler32-zlib) [12533.491725] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [12533.792094] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [12535.567299] Key type lgssc registered [12536.939636] Lustre: Echo OBD driver; http://www.lustre.org/ [12564.769794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12564.809983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12566.110781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12566.167593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12566.307256] Lustre: lustre-MDT0000: new disk, initializing [12566.441589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12566.483633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12571.486740] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12582.446244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12582.592372] Lustre: 443834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12582.668863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12582.674596] Lustre: Skipped 1 previous similar message [12582.777848] Lustre: lustre-MDT0001: new disk, initializing [12582.865732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12582.922178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12582.934159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12588.044632] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12598.408765] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12604.227540] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12611.302883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12611.632813] Lustre: lustre-OST0000: new disk, initializing [12611.642561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12611.655311] Lustre: 445802:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12611.749354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12618.397338] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12619.904869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12619.924588] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12620.087210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12628.744826] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12635.619582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12635.642439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12635.649367] Lustre: Skipped 1 previous similar message [12636.643041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12636.644313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12636.658620] Lustre: Skipped 1 previous similar message [12636.677470] Lustre: Skipped 2 previous similar messages [12641.761321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12641.771516] Lustre: Skipped 2 previous similar messages [12642.029458] Lustre: server umount lustre-MDT0000 complete [12645.686967] LustreError: 443826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624801 with bad export cookie 9198612434984673978 [12645.689305] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12645.698909] LustreError: 443826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12646.208212] Lustre: server umount lustre-MDT0001 complete [12660.439405] Lustre: server umount lustre-OST0000 complete [12670.549352] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:13:43 (1788624823) [12678.214437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12679.017921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12683.178951] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12684.258153] LustreError: 447834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12684.279112] LustreError: 447834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12688.360780] LustreError: 447835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12692.017146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12692.449682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12697.909817] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12708.175914] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12713.352694] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12733.419268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12733.618676] Lustre: Found index 0 for lustre-OST0000, updating log [12733.887234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12740.658327] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [12749.310849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12750.303601] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12758.505781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12758.523562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12758.554282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12761.664119] Lustre: server umount lustre-OST0000 complete [12779.489528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12779.499378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12779.514562] Lustre: Skipped 1 previous similar message [12779.520972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12779.525708] Lustre: Skipped 3 previous similar messages [12792.288955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12792.302791] Lustre: Skipped 7 previous similar messages [12793.313320] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12793.633515] Lustre: server umount lustre-MDT0000 complete [12797.409396] LustreError: 447829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12797.443403] LustreError: 447829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12797.786520] LustreError: 450680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624953 with bad export cookie 9198612434984675371 [12797.794750] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12797.804585] LustreError: 450680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12797.986233] LNet: 1 local NIs in recovery (showing 1): 192.168.203.119@tcp [12798.303565] Lustre: server umount lustre-MDT0001 complete [12807.554578] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:16:00 (1788624960) [13424.661392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13425.259069] LustreError: 454245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13425.347104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13430.216334] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [13430.755413] LustreError: 454246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13435.875124] LustreError: 454245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13439.597761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13440.214777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13444.270216] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [13452.225372] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13457.621888] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13478.010724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13478.266760] Lustre: Found index 0 for lustre-OST0000, updating log [13478.653920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13484.851563] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [13488.162675] LNet: 1 local NIs in recovery (showing 1): 192.168.203.119@tcp [13492.743216] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13495.133920] Lustre: server umount lustre-OST0000 complete [13499.875380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13499.880604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13499.888267] Lustre: Skipped 2 previous similar messages [13501.985050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13501.993743] Lustre: Skipped 4 previous similar messages [13506.084094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13506.102248] Lustre: Skipped 7 previous similar messages [13513.183121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13513.442863] Lustre: server umount lustre-MDT0000 complete [13514.275718] LustreError: 454246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13514.300568] LustreError: 454246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13517.602243] LustreError: 454227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625673 with bad export cookie 9198612434984676456 [13517.604489] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13517.619060] LustreError: 454227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13517.994710] Lustre: server umount lustre-MDT0001 complete [13549.254544] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [13559.735658] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [13595.626594] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:29:08 (1788625748) [13611.321761] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [13613.956836] Key type lgssc unregistered [13614.335949] LNet: 460859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13614.341307] LNetError: 460859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13739.304469] LNet: Removed LNI 192.168.203.119@tcp [13740.361352] Key type .llcrypt unregistered [13740.363639] Key type ._llcrypt unregistered [13758.605667] Key type ._llcrypt registered [13758.608108] Key type .llcrypt registered [13758.761707] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [13760.083041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13760.248626] alg: No test for adler32 (adler32-zlib) [13761.553701] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [13761.898810] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [13763.575165] Key type lgssc registered [13764.819676] Lustre: Echo OBD driver; http://www.lustre.org/ [13769.502431] LNet: Added LNI 192.168.203.119@tcp100 [8/256/0/180] [13770.401280] LNet: Added LNI 192.168.203.119@tcp101 [8/256/0/180] [13771.263905] LNet: Added LNI 192.168.203.119@tcp102 [8/256/0/180] [13772.070339] LNet: Added LNI 192.168.203.119@tcp103 [8/256/0/180] [13772.931858] LNet: Added LNI 192.168.203.119@tcp104 [8/256/0/180] [13773.782933] LNet: Added LNI 192.168.203.119@tcp105 [8/256/0/180] [13774.586454] LNet: Added LNI 192.168.203.119@tcp106 [8/256/0/180] [13775.343367] LNet: Added LNI 192.168.203.119@tcp107 [8/256/0/180] [13776.221275] LNet: Added LNI 192.168.203.119@tcp108 [8/256/0/180] [13777.128546] LNet: Added LNI 192.168.203.119@tcp109 [8/256/0/180] [13777.961068] LNet: Added LNI 192.168.203.119@tcp110 [8/256/0/180] [13778.635926] LNet: Added LNI 192.168.203.119@tcp111 [8/256/0/180] [13779.358405] LNet: Added LNI 192.168.203.119@tcp112 [8/256/0/180] [13780.050546] LNet: Added LNI 192.168.203.119@tcp113 [8/256/0/180] [13780.873740] LNet: Added LNI 192.168.203.119@tcp114 [8/256/0/180] [13781.712608] LNet: Added LNI 192.168.203.119@tcp115 [8/256/0/180] [13782.453509] LNet: Added LNI 192.168.203.119@tcp116 [8/256/0/180] [13783.202984] LNet: Added LNI 192.168.203.119@tcp117 [8/256/0/180] [13784.246148] LNet: Added LNI 192.168.203.119@tcp118 [8/256/0/180] [13785.142631] LNet: Added LNI 192.168.203.119@tcp119 [8/256/0/180] [13786.095438] LNet: Added LNI 192.168.203.119@tcp120 [8/256/0/180] [13787.011719] LNet: Added LNI 192.168.203.119@tcp121 [8/256/0/180] [13787.987724] LNet: Added LNI 192.168.203.119@tcp122 [8/256/0/180] [13789.015825] LNet: Added LNI 192.168.203.119@tcp123 [8/256/0/180] [13790.339594] LNet: Added LNI 192.168.203.119@tcp124 [8/256/0/180] [13791.408449] LNet: Added LNI 192.168.203.119@tcp125 [8/256/0/180] [13792.561696] LNet: Added LNI 192.168.203.119@tcp126 [8/256/0/180] [13793.563788] LNet: Added LNI 192.168.203.119@tcp127 [8/256/0/180] [13794.413813] LNet: Added LNI 192.168.203.119@tcp128 [8/256/0/180] [13795.260765] LNet: Added LNI 192.168.203.119@tcp129 [8/256/0/180] [13796.258268] LNet: Added LNI 192.168.203.119@tcp130 [8/256/0/180] [13797.214487] LNet: Added LNI 192.168.203.119@tcp131 [8/256/0/180] [13798.338834] LNet: Added LNI 192.168.203.119@tcp132 [8/256/0/180] [13799.271354] LNet: Added LNI 192.168.203.119@tcp133 [8/256/0/180] [13800.238890] LNet: Added LNI 192.168.203.119@tcp134 [8/256/0/180] [13800.993462] LNet: Added LNI 192.168.203.119@tcp135 [8/256/0/180] [13807.776699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13807.798790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13808.944653] Lustre: 463888:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.119@tcp: can't add NID 192.168.203.119@tcp130: rc = -75 [13809.110714] LNetError: 461347:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.119 [13809.142278] LNetError: Protocol error connecting to 192.168.203.119@tcp120 at host 192.168.203.119:988: is it running a compatible version of Lustre? [13809.161233] LNetError: Skipped 5 previous similar messages [13809.326996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13809.403886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13809.602984] Lustre: lustre-MDT0000: new disk, initializing [13809.902861] Lustre: 463945:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.119@tcp: can't add NID 192.168.203.119@tcp130: rc = -75 [13809.930988] Lustre: 463945:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13809.956911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13810.001666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13810.015582] LNet: 7 local NIs in recovery (showing 7): 192.168.203.119@tcp124, 192.168.203.119@tcp125, 192.168.203.119@tcp126, 192.168.203.119@tcp127, 192.168.203.119@tcp128, 192.168.203.119@tcp129, 192.168.203.119@tcp120 [13810.027462] LNetError: 463908:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.119 [13810.054435] LNetError: 463908:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [13810.083484] LNetError: Protocol error connecting to 192.168.203.119@tcp104 at host 192.168.203.119:988: is it running a compatible version of Lustre? [13810.097924] LNetError: Skipped 2 previous similar messages [13811.104054] LNetError: 461346:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.119 [13811.122177] LNetError: 461346:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13811.127926] LNetError: Protocol error connecting to 192.168.203.119@tcp127 at host 192.168.203.119:988: is it running a compatible version of Lustre? [13814.694485] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [13824.825288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13824.970481] Lustre: 463904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13825.045620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13825.049586] Lustre: Skipped 1 previous similar message [13825.128592] Lustre: lustre-MDT0001: new disk, initializing [13825.189137] Lustre: 464708:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.119@tcp: can't add NID 192.168.203.119@tcp130: rc = -75 [13825.205932] Lustre: 464708:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13825.359239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13825.417282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13825.440597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13829.406918] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [13831.194982] Lustre: 465072:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.119@tcp: can't add NID 192.168.203.119@tcp130: rc = -75 [13831.199974] Lustre: 465072:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13838.860537] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13840.738902] LNetError: 463912:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.119 [13840.746962] LNetError: Protocol error connecting to 192.168.203.119@tcp123 at host 192.168.203.119:988: is it running a compatible version of Lustre? [13844.010500] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13847.907746] LNetError: 463909:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.119 [13847.922421] LNetError: 463909:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13847.927730] LNetError: Protocol error connecting to 192.168.203.119@tcp127 at host 192.168.203.119:988: is it running a compatible version of Lustre? [13847.941734] LNetError: Skipped 1 previous similar message [13850.369433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13850.774137] Lustre: lustre-OST0000: new disk, initializing [13850.779924] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13850.791576] Lustre: 465882:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13850.952559] Lustre: 465887:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.119@tcp: can't add NID 192.168.203.119@tcp130: rc = -75 [13850.962386] Lustre: 465887:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13851.020825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13858.379485] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [13860.196630] LNetError: 461344:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.119 [13860.212236] LNetError: 461344:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [13860.222767] LNetError: Protocol error connecting to 192.168.203.119@tcp127 at host 192.168.203.119:988: is it running a compatible version of Lustre? [13860.229966] LNetError: Skipped 5 previous similar messages [13860.387246] Lustre: 466316:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.119@tcp: can't add NID 192.168.203.119@tcp130: rc = -75 [13860.400098] Lustre: 466316:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13860.472047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13860.489479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13860.704710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13869.643773] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13877.609922] LNetError: 461347:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.119 [13877.617558] LNetError: 461347:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [13877.623922] LNetError: Protocol error connecting to 192.168.203.119@tcp120 at host 192.168.203.119:988: is it running a compatible version of Lustre? [13877.634734] LNetError: Skipped 8 previous similar messages [13881.316359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13881.325117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13881.361303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13881.370094] Lustre: Skipped 1 previous similar message [13883.701202] Lustre: server umount lustre-OST0000 complete [13891.553884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13891.558351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13891.602099] Lustre: Skipped 2 previous similar messages [13891.625191] Lustre: Skipped 1 previous similar message [13896.673749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13896.688433] Lustre: Skipped 1 previous similar message [13901.794399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13901.812851] Lustre: Skipped 1 previous similar message [13902.303210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13902.577857] Lustre: server umount lustre-MDT0000 complete [13906.169448] LustreError: 463898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626061 with bad export cookie 6649290809125915027 [13906.178601] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13906.184810] LustreError: 463898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13906.604376] Lustre: server umount lustre-MDT0001 complete [13912.417430] LNetError: 461345:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.119 [13912.435674] LNetError: 461345:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [13912.454878] LNetError: Protocol error connecting to 192.168.203.119@tcp122 at host 192.168.203.119:988: is it running a compatible version of Lustre? [13912.470380] LNetError: Skipped 8 previous similar messages [13918.071100] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [13921.376454] Key type lgssc unregistered [13921.761222] LNet: 467795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13921.781436] LNetError: 467795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13921.812743] LNet: Removed LNI 192.168.203.119@tcp [13922.847386] LNet: Removed LNI 192.168.203.119@tcp100 [13922.853358] LNetError: 467795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13922.866307] LNetError: 467795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13922.871820] LNet: Removed LNI 192.168.203.119@tcp101 [13923.935399] LNet: Removed LNI 192.168.203.119@tcp102 [13923.944041] LNetError: 467795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13923.951875] LNetError: 467795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13923.965752] LNet: Removed LNI 192.168.203.119@tcp103 [13925.027335] LNet: Removed LNI 192.168.203.119@tcp104 [13926.047757] LNet: Removed LNI 192.168.203.119@tcp105 [13926.059042] LNetError: 467795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13926.071858] LNetError: 467795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13926.085016] LNet: Removed LNI 192.168.203.119@tcp106 [13927.135213] LNet: Removed LNI 192.168.203.119@tcp107 [13928.159483] LNet: Removed LNI 192.168.203.119@tcp108 [13929.247214] LNet: Removed LNI 192.168.203.119@tcp109 [13930.271153] LNet: Removed LNI 192.168.203.119@tcp110 [13930.275669] LNetError: 467795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13930.287382] LNetError: 467795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13930.316566] LNet: Removed LNI 192.168.203.119@tcp111 [13931.359179] LNet: Removed LNI 192.168.203.119@tcp112 [13932.383147] LNet: Removed LNI 192.168.203.119@tcp113 [13933.407224] LNet: Removed LNI 192.168.203.119@tcp114 [13934.431229] LNet: Removed LNI 192.168.203.119@tcp115 [13935.455153] LNet: Removed LNI 192.168.203.119@tcp116 [13936.479131] LNet: Removed LNI 192.168.203.119@tcp117 [13937.503151] LNet: Removed LNI 192.168.203.119@tcp118 [13938.527259] LNet: Removed LNI 192.168.203.119@tcp119 [13938.532515] LNetError: 467795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13938.538888] LNetError: 467795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13938.544377] LNet: Removed LNI 192.168.203.119@tcp120 [13939.615147] LNet: Removed LNI 192.168.203.119@tcp121 [13940.639988] LNet: Removed LNI 192.168.203.119@tcp122 [13941.663187] LNet: Removed LNI 192.168.203.119@tcp123 [13942.687161] LNet: Removed LNI 192.168.203.119@tcp124 [13943.775205] LNet: Removed LNI 192.168.203.119@tcp125 [13944.804469] LNet: Removed LNI 192.168.203.119@tcp126 [13945.823872] LNet: Removed LNI 192.168.203.119@tcp127 [13946.911312] LNet: Removed LNI 192.168.203.119@tcp128 [13947.935400] LNet: Removed LNI 192.168.203.119@tcp129 [13947.938650] LNet: Removed LNI 192.168.203.119@tcp130 [13947.945185] LNet: Removed LNI 192.168.203.119@tcp131 [13947.955885] LNet: Removed LNI 192.168.203.119@tcp132 [13947.965660] LNet: Removed LNI 192.168.203.119@tcp133 [13947.976832] LNet: Removed LNI 192.168.203.119@tcp134 [13947.994120] LNet: Removed LNI 192.168.203.119@tcp135 [13948.998349] Key type .llcrypt unregistered [13949.003847] Key type ._llcrypt unregistered [13974.135289] Key type ._llcrypt registered [13974.136731] Key type .llcrypt registered [13974.373230] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [13991.666862] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [13993.100761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13993.194084] alg: No test for adler32 (adler32-zlib) [13994.253596] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [13994.489531] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [13996.159207] Key type lgssc registered [13997.360953] Lustre: Echo OBD driver; http://www.lustre.org/ [14033.308417] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:36:26 (1788626186) [14052.558754] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [14055.591890] Key type lgssc unregistered [14055.930764] LNet: 471878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14055.957745] LNetError: 471878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14055.988811] LNet: Removed LNI 192.168.203.119@tcp [14056.915645] Key type .llcrypt unregistered [14056.919271] Key type ._llcrypt unregistered [14075.130543] Key type ._llcrypt registered [14075.133616] Key type .llcrypt registered [14075.296687] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [14076.301926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14076.359150] alg: No test for adler32 (adler32-zlib) [14077.661141] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [14078.028168] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [14079.767169] Key type lgssc registered [14081.001387] Lustre: Echo OBD driver; http://www.lustre.org/ [14090.309960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14090.367461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14091.646393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14091.694859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14091.802528] Lustre: lustre-MDT0000: new disk, initializing [14091.927983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14091.973392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14096.655925] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [14107.107166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14107.269930] Lustre: 473089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14107.341974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14107.354956] Lustre: Skipped 1 previous similar message [14107.471320] Lustre: lustre-MDT0001: new disk, initializing [14107.577728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14107.630774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14107.640951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14111.756877] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [14121.312393] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14127.367751] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14134.269640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14134.531868] Lustre: lustre-OST0000: new disk, initializing [14134.538519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14134.545205] Lustre: 475057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14134.626500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14139.895651] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [14144.092606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14144.107553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14144.227466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14150.742797] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14155.660557] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing lnet_if_list [14161.436405] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14161.457327] LNet: Added LNI 192.168.203.119@tcp10 [8/256/0/180] [14167.882771] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14167.899840] LNet: Added LNI 192.168.203.119@tcp11 [8/256/0/180] [14173.047079] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14173.072822] LNet: Added LNI 192.168.203.119@tcp12 [8/256/0/180] [14178.438456] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14178.496188] LNet: Added LNI 192.168.203.119@tcp13 [8/256/0/180] [14184.349822] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14184.409679] LNet: Added LNI 192.168.203.119@tcp14 [8/256/0/180] [14190.344426] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14190.375043] LNet: Added LNI 192.168.203.119@tcp15 [8/256/0/180] [14196.681460] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14196.718051] LNet: Added LNI 192.168.203.119@tcp16 [8/256/0/180] [14202.213403] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14202.250338] LNet: Added LNI 192.168.203.119@tcp17 [8/256/0/180] [14207.920484] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14207.940881] LNet: Added LNI 192.168.203.119@tcp18 [8/256/0/180] [14213.731745] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14213.784321] LNet: Added LNI 192.168.203.119@tcp19 [8/256/0/180] [14220.076132] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14220.108479] LNet: Added LNI 192.168.203.119@tcp20 [8/256/0/180] [14226.376966] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14226.418902] LNet: Added LNI 192.168.203.119@tcp21 [8/256/0/180] [14233.022311] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14233.054948] LNet: Added LNI 192.168.203.119@tcp22 [8/256/0/180] [14239.455382] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14239.499731] LNet: Added LNI 192.168.203.119@tcp23 [8/256/0/180] [14245.427189] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14245.452331] LNet: Added LNI 192.168.203.119@tcp24 [8/256/0/180] [14251.760424] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14251.792939] LNet: Added LNI 192.168.203.119@tcp25 [8/256/0/180] [14258.767417] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14258.809028] LNet: Added LNI 192.168.203.119@tcp26 [8/256/0/180] [14265.754796] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14265.784273] LNet: Added LNI 192.168.203.119@tcp27 [8/256/0/180] [14271.996961] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14272.058036] LNet: Added LNI 192.168.203.119@tcp28 [8/256/0/180] [14279.192504] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14279.225798] LNet: Added LNI 192.168.203.119@tcp29 [8/256/0/180] [14285.384391] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14285.425777] LNet: Added LNI 192.168.203.119@tcp30 [8/256/0/180] [14293.240820] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14293.317592] LNet: Added LNI 192.168.203.119@tcp31 [8/256/0/180] [14302.087052] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14302.136579] LNet: Added LNI 192.168.203.119@tcp32 [8/256/0/180] [14310.715033] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14310.760123] LNet: Added LNI 192.168.203.119@tcp33 [8/256/0/180] [14318.621881] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14318.653536] LNet: Added LNI 192.168.203.119@tcp34 [8/256/0/180] [14326.888625] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14326.928499] LNet: Added LNI 192.168.203.119@tcp35 [8/256/0/180] [14334.848156] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14334.898278] LNet: Added LNI 192.168.203.119@tcp36 [8/256/0/180] [14342.405344] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14342.443844] LNet: Added LNI 192.168.203.119@tcp37 [8/256/0/180] [14350.352297] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14350.387914] LNet: Added LNI 192.168.203.119@tcp38 [8/256/0/180] [14358.847440] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14358.878184] LNet: Added LNI 192.168.203.119@tcp39 [8/256/0/180] [14367.005306] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14367.064492] LNet: Added LNI 192.168.203.119@tcp40 [8/256/0/180] [14375.555953] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14375.587837] LNet: Added LNI 192.168.203.119@tcp41 [8/256/0/180] [14383.922712] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14383.955656] LNet: Added LNI 192.168.203.119@tcp42 [8/256/0/180] [14392.730276] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14392.776067] LNet: Added LNI 192.168.203.119@tcp43 [8/256/0/180] [14401.730801] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14401.773869] LNet: Added LNI 192.168.203.119@tcp44 [8/256/0/180] [14411.191166] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14411.233103] LNet: Added LNI 192.168.203.119@tcp45 [8/256/0/180] [14420.554824] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14420.619351] LNet: Added LNI 192.168.203.119@tcp46 [8/256/0/180] [14429.147942] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14429.189969] LNet: Added LNI 192.168.203.119@tcp47 [8/256/0/180] [14438.067934] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14438.124731] LNet: Added LNI 192.168.203.119@tcp48 [8/256/0/180] [14448.467768] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14448.486510] LNet: Added LNI 192.168.203.119@tcp49 [8/256/0/180] [14459.528587] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14459.585237] LNet: Added LNI 192.168.203.119@tcp50 [8/256/0/180] [14481.378593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14481.393526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14481.406675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14482.412086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14482.429963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14482.441530] Lustre: Skipped 1 previous similar message [14482.457477] Lustre: Skipped 2 previous similar messages [14487.534265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14487.546176] Lustre: Skipped 2 previous similar messages [14492.640931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14492.654300] Lustre: Skipped 2 previous similar messages [14493.151142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14493.487610] Lustre: server umount lustre-MDT0000 complete [14497.662604] LustreError: 473081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626653 with bad export cookie 9146454872383790651 [14497.665709] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14497.679197] LustreError: 473081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14497.767103] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14497.790404] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14498.187109] Lustre: server umount lustre-MDT0001 complete [14520.410594] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [14529.937705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14543.063508] LustreError: 475051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14543.090555] LustreError: 475051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14543.122617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14547.337873] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [14548.456378] LustreError: 475052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14552.546238] LustreError: 475051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14552.558720] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14556.547890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14556.941845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14562.500663] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [14563.302672] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14570.765796] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14576.596692] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14593.013361] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14593.036176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14593.055810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14598.930785] Lustre: server umount lustre-OST0000 complete [14603.232517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14603.244138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14603.251429] Lustre: Skipped 1 previous similar message [14608.415170] Lustre: server umount lustre-MDT0000 complete [14609.377025] LustreError: 499312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14609.392793] LustreError: 499312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14612.737107] LustreError: 473079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626768 with bad export cookie 9146454872383793255 [14612.749250] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14612.759561] LustreError: 473079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14613.139389] Lustre: server umount lustre-MDT0001 complete [14625.212844] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [14629.079344] Key type lgssc unregistered [14629.692891] LNet: 502088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14629.696949] LNetError: 502088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14629.711072] LNet: Removed LNI 192.168.203.119@tcp [14629.713563] LNet: Removed LNI 192.168.203.119@tcp10 [14629.722254] LNet: Removed LNI 192.168.203.119@tcp11 [14629.737759] LNet: Removed LNI 192.168.203.119@tcp12 [14629.745123] LNet: Removed LNI 192.168.203.119@tcp13 [14629.747072] LNet: Removed LNI 192.168.203.119@tcp14 [14629.752614] LNet: Removed LNI 192.168.203.119@tcp15 [14629.763611] LNet: Removed LNI 192.168.203.119@tcp16 [14629.776875] LNet: Removed LNI 192.168.203.119@tcp17 [14629.783474] LNet: Removed LNI 192.168.203.119@tcp18 [14629.789259] LNet: Removed LNI 192.168.203.119@tcp19 [14629.793813] LNet: Removed LNI 192.168.203.119@tcp20 [14629.796995] LNet: Removed LNI 192.168.203.119@tcp21 [14629.800323] LNet: Removed LNI 192.168.203.119@tcp22 [14629.805418] LNet: Removed LNI 192.168.203.119@tcp23 [14629.808986] LNet: Removed LNI 192.168.203.119@tcp24 [14629.812577] LNet: Removed LNI 192.168.203.119@tcp25 [14629.814298] LNet: Removed LNI 192.168.203.119@tcp26 [14629.817471] LNet: Removed LNI 192.168.203.119@tcp27 [14629.821443] LNet: Removed LNI 192.168.203.119@tcp28 [14629.824706] LNet: Removed LNI 192.168.203.119@tcp29 [14629.827840] LNet: Removed LNI 192.168.203.119@tcp30 [14629.830923] LNet: Removed LNI 192.168.203.119@tcp31 [14629.834322] LNet: Removed LNI 192.168.203.119@tcp32 [14629.837313] LNet: Removed LNI 192.168.203.119@tcp33 [14629.839925] LNet: Removed LNI 192.168.203.119@tcp34 [14629.843395] LNet: Removed LNI 192.168.203.119@tcp35 [14629.846748] LNet: Removed LNI 192.168.203.119@tcp36 [14629.849807] LNet: Removed LNI 192.168.203.119@tcp37 [14629.852088] LNet: Removed LNI 192.168.203.119@tcp38 [14629.854712] LNet: Removed LNI 192.168.203.119@tcp39 [14629.857935] LNet: Removed LNI 192.168.203.119@tcp40 [14629.860924] LNet: Removed LNI 192.168.203.119@tcp41 [14629.864008] LNet: Removed LNI 192.168.203.119@tcp42 [14629.866960] LNet: Removed LNI 192.168.203.119@tcp43 [14629.870132] LNet: Removed LNI 192.168.203.119@tcp44 [14629.873367] LNet: Removed LNI 192.168.203.119@tcp45 [14629.875968] LNet: Removed LNI 192.168.203.119@tcp46 [14629.879314] LNet: Removed LNI 192.168.203.119@tcp47 [14629.882420] LNet: Removed LNI 192.168.203.119@tcp48 [14629.885272] LNet: Removed LNI 192.168.203.119@tcp49 [14629.910135] LNet: Removed LNI 192.168.203.119@tcp50 [14630.850373] Key type .llcrypt unregistered [14630.852594] Key type ._llcrypt unregistered [14656.331767] Key type ._llcrypt registered [14656.338113] Key type .llcrypt registered [14656.443781] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [14672.550965] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [14673.633182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14673.810484] alg: No test for adler32 (adler32-zlib) [14674.905570] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [14675.173745] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [14676.831167] Key type lgssc registered [14677.903830] Lustre: Echo OBD driver; http://www.lustre.org/ [14712.201431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14712.234386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14713.531178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14713.559734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14713.642711] Lustre: lustre-MDT0000: new disk, initializing [14713.718545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14713.737262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14718.300418] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [14729.397375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14729.633072] Lustre: 505284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14729.763410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14729.767572] Lustre: Skipped 1 previous similar message [14729.858541] Lustre: lustre-MDT0001: new disk, initializing [14729.982258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14730.016625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14730.022557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14737.819854] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [14749.963415] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14756.814351] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14764.635905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14764.788474] Lustre: *** cfs_fail_loc=90f, val=0*** [14764.917607] Lustre: lustre-OST0000: new disk, initializing [14764.925106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14764.944200] Lustre: 507301:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14765.137045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14771.808193] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [14772.334458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14772.349726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14772.485396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14782.476073] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14793.189332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14793.193661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14793.197758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14793.210729] Lustre: Skipped 1 previous similar message [14793.251832] Lustre: Skipped 1 previous similar message [14797.179383] Lustre: server umount lustre-OST0000 complete [14801.889793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14801.898744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14801.908333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14803.426822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14803.429696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14803.464790] Lustre: Skipped 1 previous similar message [14808.185902] Lustre: server umount lustre-MDT0000 complete [14812.631884] LustreError: 505276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626968 with bad export cookie 9994820197656073808 [14812.640366] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14812.645997] LustreError: 505276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14813.301680] Lustre: server umount lustre-MDT0001 complete [14826.243715] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [14829.689774] Key type lgssc unregistered [14830.076692] LNet: 509207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14830.083839] LNetError: 509207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14830.114414] LNet: Removed LNI 192.168.203.119@tcp [14831.225860] Key type .llcrypt unregistered [14831.231489] Key type ._llcrypt unregistered [14852.523330] Key type ._llcrypt registered [14852.525731] Key type .llcrypt registered [14852.675912] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [14854.356133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14854.623569] alg: No test for adler32 (adler32-zlib) [14855.716676] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [14856.102669] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [14857.911786] Key type lgssc registered [14859.268721] Lustre: Echo OBD driver; http://www.lustre.org/ [14868.103090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14868.123595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14869.806184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14874.889700] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [14880.241291] LustreError: 510476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14880.266518] LustreError: 510476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14884.142164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14884.679410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14889.346079] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [14897.357305] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14903.735634] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14911.144366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14911.460427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14918.349415] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [14926.957486] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14940.801584] loop0: detected capacity change from 0 to 204800000 [14941.060866] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14941.090019] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14941.107718] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14947.744371] loop0: detected capacity change from 0 to 204800000 [14947.909494] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14947.930087] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14947.955236] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14954.063647] loop0: detected capacity change from 0 to 204800000 [14960.951157] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14961.179349] Lustre: fs15246-OST0000: new disk, initializing [14961.188786] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14961.210946] Lustre: 514073:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14961.313857] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14968.695568] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [14979.358329] loop1: detected capacity change from 0 to 204800000 [14985.930859] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14986.094506] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14986.157626] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14986.272368] Lustre: fs15246-MDT0000: new disk, initializing [14986.539083] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14986.600416] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14986.787675] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14987.809348] Lustre: 515195:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f4a6e5f2 x1875511356735872/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788627159 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14992.234843] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [14996.914499] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14996.928028] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14996.978729] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15015.397103] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15015.426341] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15020.531287] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15021.109920] Lustre: server umount fs15246-MDT0000 complete [15025.636401] LustreError: 512919:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15030.760473] LustreError: 510500:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15031.197855] Lustre: server umount fs15246-OST0000 complete [15046.117884] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15046.132352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15046.143781] Lustre: Skipped 1 previous similar message [15046.163413] Lustre: Skipped 1 previous similar message [15051.040278] Lustre: server umount lustre-OST0000 complete [15056.358221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15056.406583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15056.422025] Lustre: Skipped 1 previous similar message [15061.475261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15061.495211] Lustre: Skipped 1 previous similar message [15062.319384] Lustre: server umount lustre-MDT0000 complete [15066.595412] LustreError: 511205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15066.635245] LustreError: 511205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15066.662978] LustreError: 512320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627222 with bad export cookie 14430180207466658871 [15066.671200] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15066.688671] LustreError: 512320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15067.234578] Lustre: server umount lustre-MDT0001 complete [15080.256466] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [15083.471930] Key type lgssc unregistered [15083.825050] LNet: 518292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15083.856911] LNetError: 518292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15083.885915] LNet: Removed LNI 192.168.203.119@tcp [15084.902740] Key type .llcrypt unregistered [15084.904819] Key type ._llcrypt unregistered [15105.673428] Key type ._llcrypt registered [15105.675755] Key type .llcrypt registered [15105.874933] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [15107.501129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15107.572772] alg: No test for adler32 (adler32-zlib) [15108.752350] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [15109.025214] LNet: Added LNI 192.168.203.119@tcp [8/256/0/180] [15110.759382] Key type lgssc registered [15112.403204] Lustre: Echo OBD driver; http://www.lustre.org/ [15153.640754] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [15166.368743] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [15201.815869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15201.860156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15203.224818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15203.253567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15203.346222] Lustre: lustre-MDT0000: new disk, initializing [15203.437566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15203.466114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15208.675765] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [15220.096231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15220.216670] Lustre: 522756:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15220.303329] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15220.305219] Lustre: Skipped 1 previous similar message [15220.432477] Lustre: lustre-MDT0001: new disk, initializing [15220.533464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15220.584314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15220.608681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15225.863831] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [15235.937629] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15242.227098] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15248.974210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15249.211773] Lustre: lustre-OST0000: new disk, initializing [15249.221571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15249.229608] Lustre: 524726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15249.334477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15254.650897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15254.662909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15254.781516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15255.700913] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_default_debug -1 all [15264.759928] Lustre: DEBUG MARKER: oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15269.621750] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15269.921696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15275.161288] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15275.404956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15280.609640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15280.616962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15280.624774] LustreError: Skipped 1 previous similar message [15280.628871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15280.642576] Lustre: Skipped 1 previous similar message [15283.648542] Lustre: server umount lustre-OST0000 complete [15290.852099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15290.854995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15290.875271] Lustre: Skipped 1 previous similar message [15290.915164] Lustre: Skipped 2 previous similar messages [15294.022469] Lustre: server umount lustre-MDT0000 complete [15297.891842] LustreError: 522748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627453 with bad export cookie 7192791801704008172 [15297.895846] LustreError: MGC192.168.203.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15297.907017] LustreError: 522748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15298.244800] Lustre: server umount lustre-MDT0001 complete [15323.655276] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing set_hostid [15335.352652] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing load_modules_local [15369.797417] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15106 sec ========= 12:58:42 (1788627522) [15371.594790] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:58:44 (1788627524) === [15375.889281] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:58:48 (1788627528) === [15417.534105] Lustre: DEBUG MARKER: oleg319-server.virtnet: executing unload_modules_local [15420.400091] Key type lgssc unregistered [15420.736624] LNet: 531372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15420.762737] LNetError: 531372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15421.808224] LNet: Removed LNI 192.168.203.119@tcp [15422.741896] Key type .llcrypt unregistered [15422.744186] Key type ._llcrypt unregistered