[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 499715877 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003212] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008593] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011157] LSM: Security Framework initializing [ 0.012063] Yama: becoming mindful. [ 0.013041] SELinux: Initializing. [ 0.014070] *** VALIDATE selinux *** [ 0.022404] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027225] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030151] *** VALIDATE tmpfs *** [ 0.032041] *** VALIDATE proc *** [ 0.033230] *** VALIDATE cgroup *** [ 0.034007] *** VALIDATE cgroup2 *** [ 0.035234] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037016] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039001] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043011] debug: unmapping init [mem 0xffffffff93a59000-0xffffffff93a60fff] [ 0.045182] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046718] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051011] ... max period: 00007fffffffffff [ 0.052012] ... fixed-purpose events: 3 [ 0.053010] ... event mask: 000000070000000f [ 0.054260] rcu: Hierarchical SRCU implementation. [ 0.056324] smp: Bringing up secondary CPUs ... [ 0.057495] x86: Booting SMP configuration: [ 0.058026] .... node #0, CPUs: #1 #2 #3 [ 0.061273] smp: Brought up 1 node, 4 CPUs [ 0.062891] smpboot: Max logical packages: 1 [ 0.063011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.204679] node 0 deferred pages initialised in 139ms [ 0.206092] devtmpfs: initialized [ 0.207313] x86/mm: Memory block size: 128MB [ 0.211089] gcov: version magic: 0x41383552 [ 0.214322] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.218114] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.220427] pinctrl core: initialized pinctrl subsystem [ 0.223303] [ 0.224000] ************************************************************* [ 0.226019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.228017] ** ** [ 0.231021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.233017] ** ** [ 0.236017] ** This means that this kernel is built to expose internal ** [ 0.238015] ** IOMMU data structures, which may compromise security on ** [ 0.240014] ** your system. ** [ 0.241011] ** ** [ 0.243015] ** If you see this message and you are not debugging the ** [ 0.244012] ** kernel, report this immediately to your vendor! ** [ 0.246013] ** ** [ 0.248024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.251019] ************************************************************* [ 0.254027] NET: Registered protocol family 16 [ 0.256616] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.259088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.262093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.265251] cpuidle: using governor menu [ 0.266604] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.267636] PCI: Using configuration type 1 for base access [ 0.270125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.279066] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.281051] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.285119] cryptd: max_cpu_qlen set to 1000 [ 0.288276] ACPI: Added _OSI(Module Device) [ 0.290024] ACPI: Added _OSI(Processor Device) [ 0.291018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.293022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.298038] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.304567] ACPI: Interpreter enabled [ 0.306076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.308022] ACPI: Using IOAPIC for interrupt routing [ 0.310146] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.313442] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.323845] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.326061] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.329029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.333117] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.338471] acpiphp: Slot [2] registered [ 0.340273] acpiphp: Slot [5] registered [ 0.342218] acpiphp: Slot [6] registered [ 0.343168] acpiphp: Slot [7] registered [ 0.345191] acpiphp: Slot [8] registered [ 0.346163] acpiphp: Slot [9] registered [ 0.348200] acpiphp: Slot [10] registered [ 0.349188] acpiphp: Slot [3] registered [ 0.351145] acpiphp: Slot [4] registered [ 0.353145] acpiphp: Slot [11] registered [ 0.354120] acpiphp: Slot [12] registered [ 0.356182] acpiphp: Slot [13] registered [ 0.357137] acpiphp: Slot [14] registered [ 0.359175] acpiphp: Slot [15] registered [ 0.360128] acpiphp: Slot [16] registered [ 0.362130] acpiphp: Slot [17] registered [ 0.364176] acpiphp: Slot [18] registered [ 0.365144] acpiphp: Slot [19] registered [ 0.367155] acpiphp: Slot [20] registered [ 0.369147] acpiphp: Slot [21] registered [ 0.370195] acpiphp: Slot [22] registered [ 0.372146] acpiphp: Slot [23] registered [ 0.374132] acpiphp: Slot [24] registered [ 0.376103] acpiphp: Slot [25] registered [ 0.377162] acpiphp: Slot [26] registered [ 0.378150] acpiphp: Slot [27] registered [ 0.380142] acpiphp: Slot [28] registered [ 0.382148] acpiphp: Slot [29] registered [ 0.383145] acpiphp: Slot [30] registered [ 0.385194] acpiphp: Slot [31] registered [ 0.386108] PCI host bridge to bus 0000:00 [ 0.388040] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.391035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.393042] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.396035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.398038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.400035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.401266] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.405051] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.408475] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.418024] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.422552] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.425030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.428028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.430024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.433692] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.436854] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.439063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.442453] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.445956] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.458023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.461987] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.468244] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.478021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.489027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.506022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.514707] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.520023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.526026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.542019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.553885] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.559018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.564025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.577022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.586544] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.592025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.597024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.613022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.622824] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.630018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.636018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.650020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.660268] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.668025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.679033] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.700103] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.713612] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.716377] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.718373] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.720422] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.723279] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.728764] iommu: Default domain type: Passthrough [ 0.730507] SCSI subsystem initialized [ 0.732135] ACPI: bus type USB registered [ 0.734139] usbcore: registered new interface driver usbfs [ 0.736090] usbcore: registered new interface driver hub [ 0.738094] usbcore: registered new device driver usb [ 0.740209] pps_core: LinuxPPS API ver. 1 registered [ 0.742015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.745062] PTP clock support registered [ 0.747212] EDAC MC: Ver: 3.0.0 [ 0.749148] PCI: Using ACPI for IRQ routing [ 0.750478] NetLabel: Initializing [ 0.752028] NetLabel: domain hash size = 128 [ 0.753012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.755087] NetLabel: unlabeled traffic allowed by default [ 0.758114] vgaarb: loaded [ 0.759360] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.761019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.768013] clocksource: Switched to clocksource kvm-clock [ 0.866900] VFS: Disk quotas dquot_6.6.0 [ 0.868548] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.871140] *** VALIDATE ramfs *** [ 0.872456] *** VALIDATE hugetlbfs *** [ 0.874140] pnp: PnP ACPI init [ 0.876579] pnp: PnP ACPI: found 6 devices [ 0.900084] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.903478] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.905631] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.907764] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.910128] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.912456] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.915611] NET: Registered protocol family 2 [ 0.918075] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.922957] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.926739] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.932908] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.936868] TCP: Hash tables configured (established 65536 bind 65536) [ 0.940212] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.943549] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.946643] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.949751] NET: Registered protocol family 1 [ 0.952297] RPC: Registered named UNIX socket transport module. [ 0.954073] RPC: Registered udp transport module. [ 0.955614] RPC: Registered tcp transport module. [ 0.957643] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.960483] NET: Registered protocol family 44 [ 0.962328] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.964521] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.966557] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.968913] PCI: CLS 0 bytes, default 64 [ 0.971233] Unpacking initramfs... [ 2.367913] debug: unmapping init [mem 0xffff97df7cc54000-0xffff97df7ffbffff] [ 2.371591] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.373963] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.376977] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.875478] Initialise system trusted keyrings [ 2.877211] Key type blacklist registered [ 2.879072] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.888512] zbud: loaded [ 2.892050] *** VALIDATE nfs *** [ 2.893256] *** VALIDATE nfs4 *** [ 2.894528] pstore: using deflate compression [ 2.897737] Platform Keyring initialized [ 3.016613] NET: Registered protocol family 38 [ 3.018851] Key type asymmetric registered [ 3.020611] Asymmetric key parser 'x509' registered [ 3.022989] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.026355] io scheduler mq-deadline registered [ 3.027924] io scheduler kyber registered [ 3.029264] io scheduler bfq registered [ 3.030711] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.034096] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.036365] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.040174] ACPI: Power Button [PWRF] [ 3.046121] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.052977] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.069862] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.078857] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.096360] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.124344] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.152357] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.157462] Non-volatile memory driver v1.3 [ 3.160699] Linux agpgart interface v0.103 [ 3.198743] virtio_blk virtio1: [vda] 146608 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.201759] vda: detected capacity change from 0 to 75063296 [ 3.217791] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.221048] vdb: detected capacity change from 0 to 1073741824 [ 3.237258] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.240244] vdc: detected capacity change from 0 to 2621440000 [ 3.255692] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.257638] vdd: detected capacity change from 0 to 2621440000 [ 3.269769] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.271583] vde: detected capacity change from 0 to 4294967296 [ 3.282081] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.284127] vdf: detected capacity change from 0 to 4294967296 [ 3.288910] libphy: Fixed MDIO Bus: probed [ 3.294215] usbcore: registered new interface driver usbserial_generic [ 3.296425] usbserial: USB Serial support registered for generic [ 3.298580] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.302394] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.304320] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.306945] mousedev: PS/2 mouse device common for all mice [ 3.309232] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.311531] rtc_cmos 00:05: RTC can wake from S4 [ 3.314595] rtc_cmos 00:05: registered as rtc0 [ 3.315681] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.315943] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.321548] intel_pstate: CPU model not supported [ 3.322989] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.326907] hid: raw HID events driver (C) Jiri Kosina [ 3.328727] usbcore: registered new interface driver usbhid [ 3.330737] usbhid: USB HID core driver [ 3.332501] drop_monitor: Initializing network drop monitor service [ 3.334838] Initializing XFRM netlink socket [ 3.336944] NET: Registered protocol family 10 [ 3.340495] Segment Routing with IPv6 [ 3.342101] NET: Registered protocol family 17 [ 3.344421] mpls_gso: MPLS GSO support [ 3.349186] RAS: Correctable Errors collector initialized. [ 3.350656] AVX version of gcm_enc/dec engaged. [ 3.351598] AES CTR mode by8 optimization enabled [ 3.451090] sched_clock: Marking stable (3451044810, 0)->(4338800021, -887755211) [ 3.454522] registered taskstats version 1 [ 3.455811] Loading compiled-in X.509 certificates [ 3.458266] zswap: loaded using pool lzo/zbud [ 3.482308] Key type big_key registered [ 3.495064] Key type encrypted registered [ 3.496286] ima: No TPM chip found, activating TPM-bypass! [ 3.497693] ima: Allocated hash algorithm: sha1 [ 3.498843] ima: No architecture policies found [ 3.500229] evm: Initialising EVM extended attributes: [ 3.501425] evm: security.selinux [ 3.502257] evm: security.ima [ 3.503140] evm: security.capability [ 3.503987] evm: HMAC attrs: 0x1 [ 3.506101] rtc_cmos 00:05: setting system clock to 2026-08-27 00:40:20 UTC (1787791220) [ 3.510443] debug: unmapping init [mem 0xffffffff94a03000-0xffffffff94bfffff] [ 3.512453] debug: unmapping init [mem 0xffffffff93782000-0xffffffff93a58fff] [ 3.518060] Write protecting the kernel read-only data: 28672k [ 3.520708] debug: unmapping init [mem 0xffffffff91e03000-0xffffffff91ffffff] [ 3.522297] debug: unmapping init [mem 0xffffffff92714000-0xffffffff927fffff] [ 3.549336] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.556475] systemd[1]: Detected virtualization kvm. [ 3.558376] systemd[1]: Detected architecture x86-64. [ 3.560411] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.587675] systemd[1]: No hostname configured. [ 3.589607] systemd[1]: Set hostname to . [ 3.591616] random: systemd: uninitialized urandom read (16 bytes read) [ 3.594044] systemd[1]: Initializing machine ID from random generator. [ 3.639961] random: ln: uninitialized urandom read (6 bytes read) [ 3.720939] random: systemd: uninitialized urandom read (16 bytes read) [ 3.723206] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.727179] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.731371] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. Starting Setup Virtual Console... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.347120] device-mapper: uevent: version 1.0.3 [ 4.349527] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.018898] virtio_net virtio0 ens2: renamed from eth0 [ 5.047866] random: fast init done [ 5.116257] scsi host0: ata_piix [ 5.149316] scsi host1: ata_piix [ 5.171392] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.173691] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.803818] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.016684] random: crng init done [ 10.018260] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.395401] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.582408] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.816639] SELinux: Disabled at runtime. [ 11.873361] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.880246] systemd[1]: Detected virtualization kvm. [ 11.881675] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.291499] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.294130] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.297819] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.301539] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.305508] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.316524] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.321525] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ 12.379802] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.796288] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.133413] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.133862] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.334968] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.348604] EDAC sbridge: Ver: 1.1.2 [ 14.888718] Key type dns_resolver registered [ 15.180365] NFS: Registering the id_resolver key type [ 15.182394] Key type id_resolver registered [ 15.183973] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg139-server login: [ 65.870851] libcfs: loading out-of-tree module taints kernel. [ 65.942531] Key type ._llcrypt registered [ 65.947114] Key type .llcrypt registered [ 66.027112] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 81.985950] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 83.554333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 83.567320] alg: No test for adler32 (adler32-zlib) [ 84.848753] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 85.677479] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 87.399178] Key type lgssc registered [ 88.833799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.975137] hrtimer: interrupt took 2079068 ns [ 102.713779] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 139.012554] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 148.813882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 148.835335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.997256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 150.022442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 150.088389] Lustre: lustre-MDT0000: new disk, initializing [ 150.149447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 150.166220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 155.052654] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.723922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.893980] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.976952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.981873] Lustre: Skipped 1 previous similar message [ 170.075653] Lustre: lustre-MDT0001: new disk, initializing [ 170.160700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 170.204239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 170.208633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.868789] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.781495] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 188.116866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 188.348618] Lustre: lustre-OST0000: new disk, initializing [ 188.350890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 188.354920] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 188.394850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.135794] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.054096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.066778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.130844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.660450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.838819] Lustre: lustre-OST0001: new disk, initializing [ 205.842596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 205.849319] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 205.955370] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 212.031152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 212.043073] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 212.098432] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 212.321902] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.486621] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 231.324923] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 237.255963] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing check_logdir /tmp/testlogs/ [ 243.102533] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing yml_node [ 247.028383] Lustre: DEBUG MARKER: Client: 2.17.57.50 [ 249.965211] Lustre: DEBUG MARKER: MDS: 2.17.57.50 [ 252.004836] Lustre: DEBUG MARKER: OSS: 2.17.57.50 [ 253.220152] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 20:44:30 EDT 2026 [ 273.610264] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 275.233578] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 293.345710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 293.349320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.373872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 294.367947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 294.370594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.374209] Lustre: Skipped 1 previous similar message [ 294.383830] Lustre: Skipped 1 previous similar message [ 298.234424] Lustre: server umount lustre-MDT0000 complete [ 301.416518] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791518 with bad export cookie 18436476817054271651 [ 301.417689] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 301.424565] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 304.611326] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.613738] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.614517] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 304.614527] Lustre: Skipped 2 previous similar messages [ 304.626486] Lustre: Skipped 2 previous similar messages [ 304.642648] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 306.659246] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 306.659944] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 306.685506] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 306.709923] Lustre: Skipped 2 previous similar messages [ 307.877251] Lustre: server umount lustre-MDT0001 complete [ 311.377922] Lustre: server umount lustre-OST0000 complete [ 314.794597] Lustre: server umount lustre-OST0001 complete [ 321.028877] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 328.934601] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 360.219277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 360.398377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 360.426692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 360.512798] Lustre: lustre-MDT0000: new disk, initializing [ 360.586507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 360.599191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 364.585915] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 373.840552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 373.940358] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 373.988989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 373.992731] Lustre: Skipped 1 previous similar message [ 374.089618] Lustre: lustre-MDT0001: new disk, initializing [ 374.196243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 374.236577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 374.246976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 378.032982] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 386.458634] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 391.131081] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 397.592786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 397.887405] Lustre: lustre-OST0000: new disk, initializing [ 397.891573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 397.896283] Lustre: 17047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 397.990217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.189476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 399.200179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 399.347090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 404.277502] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 412.845413] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 417.209833] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 417.461397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 422.147975] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 422.455344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 424.928380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 424.930105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 424.938610] LustreError: Skipped 1 previous similar message [ 424.945957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 424.959294] Lustre: Skipped 1 previous similar message [ 424.979641] Lustre: Skipped 1 previous similar message [ 430.754690] Lustre: server umount lustre-OST0000 complete [ 435.169922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 435.194479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 435.198936] Lustre: Skipped 3 previous similar messages [ 440.634921] Lustre: server umount lustre-MDT0000 complete [ 444.120558] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791661 with bad export cookie 18436476817054273184 [ 444.122072] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 444.129514] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 444.365819] Lustre: server umount lustre-MDT0001 complete [ 447.913773] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:47:44 (1787791664) [ 450.131825] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 451.830752] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:47:48 (1787791668) [ 457.928538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.358770] LustreError: 19455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 458.442208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.498924] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 463.842640] LustreError: 19456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 468.965971] LustreError: 19455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 469.409631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.482402] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 480.772822] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 485.570449] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 491.193335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 496.590403] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 502.651645] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 514.018427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 514.024044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.030545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 514.030555] Lustre: Skipped 2 previous similar messages [ 514.045676] Lustre: Skipped 2 previous similar messages [ 520.225452] Lustre: server umount lustre-OST0000 complete [ 529.866534] Lustre: server umount lustre-MDT0000 complete [ 533.106466] LustreError: 19435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791750 with bad export cookie 18436476817054274066 [ 533.108108] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.116010] LustreError: 19435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 533.431993] Lustre: server umount lustre-MDT0001 complete [ 542.936992] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 546.734412] Key type lgssc unregistered [ 547.125281] LNet: 22990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 547.132254] LNetError: 22990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 547.146595] LNet: Removed LNI 192.168.201.139@tcp [ 548.139126] Key type .llcrypt unregistered [ 548.146373] Key type ._llcrypt unregistered [ 566.393796] Key type ._llcrypt registered [ 566.395697] Key type .llcrypt registered [ 566.511512] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 567.665259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 567.717635] alg: No test for adler32 (adler32-zlib) [ 568.715172] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 568.998542] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 570.703337] Key type lgssc registered [ 572.293312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 580.553735] loop: module loaded [ 580.678070] loop0: detected capacity change from 0 to 204800000 [ 581.041099] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 581.057782] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 581.459034] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 587.451743] loop0: detected capacity change from 0 to 204800000 [ 587.701538] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 587.714990] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 587.731745] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 592.805433] loop0: detected capacity change from 0 to 204800000 [ 597.887815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 597.907812] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 599.168970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 599.227801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 599.337843] Lustre: lustre-MDT0000: new disk, initializing [ 599.456900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 599.484475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 603.649435] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 611.130215] loop1: detected capacity change from 0 to 204800000 [ 616.355261] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 616.664221] Lustre: lustre-OST07c6: new disk, initializing [ 616.667209] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 616.671724] Lustre: Skipped 1 previous similar message [ 616.676604] Lustre: 26155:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 616.786584] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 622.592499] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 626.810308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 626.821446] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 626.859216] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 632.743909] Lustre: 25143:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 632.757629] Lustre: 25143:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 632.769852] Lustre: 25143:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 632.785827] Lustre: 25143:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 632.796706] Lustre: 25143:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 632.811813] Lustre: 25143:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 649.543343] Lustre: 25143:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 649.557743] Lustre: 25143:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 649.564715] Lustre: 25143:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 649.577290] Lustre: 25143:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 649.585553] Lustre: 25143:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 649.590863] Lustre: 25143:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 653.990584] Lustre: 26841:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 654.000912] Lustre: 26841:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 654.015129] Lustre: 26841:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 654.025267] Lustre: 26841:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 654.033276] Lustre: 26841:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 654.043265] Lustre: 26841:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 654.050879] Lustre: 26841:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 654.057523] Lustre: 26841:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 654.063223] Lustre: 26841:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 654.073570] Lustre: 26841:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 654.081230] Lustre: 26841:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 654.087133] Lustre: 26841:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 657.322201] Lustre: 25142:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 657.338973] Lustre: 25142:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 657.346709] Lustre: 25142:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 657.356281] Lustre: 25142:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 657.360649] Lustre: 25142:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 657.365631] Lustre: 25142:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 657.378946] Lustre: 25142:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 657.387963] Lustre: 25142:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 657.396327] Lustre: 25142:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 657.408473] Lustre: 25142:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 657.419983] Lustre: 25142:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 657.437271] Lustre: 25142:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 662.067916] Lustre: 25144:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 662.074357] Lustre: 25144:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 662.079778] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 662.085172] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 662.090073] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 662.096702] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 662.104456] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 662.111848] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 662.117606] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 662.124456] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 662.131142] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 662.138086] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 672.993936] Lustre: 26841:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 673.004483] Lustre: 26841:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 673.008045] Lustre: 26841:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 673.013099] Lustre: 26841:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 673.018600] Lustre: 26841:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 673.028755] Lustre: 26841:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 673.037030] Lustre: 26841:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 673.045975] Lustre: 26841:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 673.050726] Lustre: 26841:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 673.058919] Lustre: 26841:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 673.064240] Lustre: 26841:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 673.071094] Lustre: 26841:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 723.936928] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 723.947259] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 723.969929] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 725.938697] Lustre: server umount lustre-OST07c6 complete [ 736.486839] Lustre: server umount lustre-MDT0000 complete [ 753.891472] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 756.625884] Key type lgssc unregistered [ 756.901433] LNet: 28718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 756.904783] LNetError: 28718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 756.917819] LNet: Removed LNI 192.168.201.139@tcp [ 757.745159] Key type .llcrypt unregistered [ 757.750551] Key type ._llcrypt unregistered [ 774.053552] Key type ._llcrypt registered [ 774.056440] Key type .llcrypt registered [ 774.196994] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 775.570751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 775.596742] alg: No test for adler32 (adler32-zlib) [ 776.820506] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 777.052330] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 778.703618] Key type lgssc registered [ 779.747647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 787.377579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 787.391190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 788.817952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 792.442505] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 798.178407] LustreError: 29969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 798.193259] LustreError: 29969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 799.991259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 800.417426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 805.115397] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 811.757226] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 816.273537] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 823.211848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 823.607898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 829.863694] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 833.843642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 837.551116] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 854.496918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 854.507801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 854.510135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 854.518793] Lustre: Skipped 1 previous similar message [ 854.533470] Lustre: Skipped 1 previous similar message [ 859.012937] Lustre: server umount lustre-OST0000 complete [ 864.739069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 864.740782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 864.749335] Lustre: Skipped 1 previous similar message [ 864.756019] Lustre: Skipped 1 previous similar message [ 869.006744] Lustre: server umount lustre-MDT0000 complete [ 869.866059] LustreError: 29970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 869.889365] LustreError: 29970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 873.358204] LustreError: 29951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792090 with bad export cookie 2986147238268956709 [ 873.368476] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 873.377148] LustreError: 29951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 873.769461] Lustre: server umount lustre-MDT0001 complete [ 885.266807] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 887.089870] Key type lgssc unregistered [ 887.350300] LNet: 33707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.361585] LNetError: 33707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 887.377560] LNet: Removed LNI 192.168.201.139@tcp [ 888.335975] Key type .llcrypt unregistered [ 888.340789] Key type ._llcrypt unregistered [ 906.059865] Key type ._llcrypt registered [ 906.067968] Key type .llcrypt registered [ 906.195422] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 907.305694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 907.346122] alg: No test for adler32 (adler32-zlib) [ 908.463322] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 908.701100] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 910.351182] Key type lgssc registered [ 911.198347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 917.884415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 917.932078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 919.309913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.397819] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 929.764826] LustreError: 34959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.778702] LustreError: 34959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 931.587408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.970538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 935.935983] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 942.703869] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 947.777785] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 956.102915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 956.531344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.499182] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 964.737668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 970.970947] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 980.451501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 980.452262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 980.465503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 980.477513] Lustre: Skipped 1 previous similar message [ 985.573416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 986.389856] Lustre: server umount lustre-OST0000 complete [ 990.690983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 990.694605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 990.704335] Lustre: Skipped 2 previous similar messages [ 990.708776] Lustre: Skipped 1 previous similar message [ 995.647353] Lustre: server umount lustre-MDT0000 complete [ 995.829621] LustreError: 34977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 995.849245] LustreError: 34977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 999.357785] LustreError: 34938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792216 with bad export cookie 16725571334905549148 [ 999.361580] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 999.363429] LustreError: 34938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 999.556082] Lustre: server umount lustre-MDT0001 complete [ 1008.505239] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1010.664429] Key type lgssc unregistered [ 1010.860675] LNet: 38494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.863995] LNetError: 38494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.876703] LNet: Removed LNI 192.168.201.139@tcp [ 1011.471306] Key type .llcrypt unregistered [ 1011.472861] Key type ._llcrypt unregistered [ 1034.373465] Key type ._llcrypt registered [ 1034.374604] Key type .llcrypt registered [ 1034.467799] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1035.640194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1035.682091] alg: No test for adler32 (adler32-zlib) [ 1036.716045] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 1037.056604] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1038.719631] Key type lgssc registered [ 1039.658755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1047.156971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1047.198685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.689735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.215719] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1058.278844] LustreError: 39911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1058.312570] LustreError: 39911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1063.394174] LustreError: 39912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.149888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1064.920493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1069.691143] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1077.021474] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1082.938899] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1089.776382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1090.013269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1093.108271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1096.241965] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1103.732218] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1114.602537] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.614301] Lustre: Skipped 1 previous similar message [ 1114.614954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.630565] Lustre: Skipped 2 previous similar messages [ 1115.775517] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) [ 1118.814447] Lustre: server umount lustre-MDT0000 complete [ 1119.728635] LustreError: 40911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1119.748104] LustreError: 40911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1122.133284] LustreError: 39892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792339 with bad export cookie 8306781451769410118 [ 1122.144121] LustreError: 39892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1122.151098] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1124.834488] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.836416] LustreError: 41269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1124.849283] Lustre: Skipped 1 previous similar message [ 1124.855797] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1124.864480] LustreError: 41269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1127.854788] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1127.860191] Lustre: Skipped 1 previous similar message [ 1128.696881] Lustre: server umount lustre-MDT0001 complete [ 1158.111700] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1158.285169] Lustre: server umount lustre-OST0000 complete [ 1176.435606] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1179.265876] Key type lgssc unregistered [ 1179.538492] LNet: 43778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.558884] LNetError: 43778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1180.595181] LNet: Removed LNI 192.168.201.139@tcp [ 1181.480138] Key type .llcrypt unregistered [ 1181.482217] Key type ._llcrypt unregistered [ 1206.591990] Key type ._llcrypt registered [ 1206.594111] Key type .llcrypt registered [ 1206.703439] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1208.116401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1208.168620] alg: No test for adler32 (adler32-zlib) [ 1209.253819] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 1209.485056] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1211.183176] Key type lgssc registered [ 1212.302425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1219.284699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1219.312873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1220.767746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1224.613478] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1230.310285] LustreError: 45197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.328130] LustreError: 45197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1231.879219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.120669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1236.350402] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1244.366647] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1250.358954] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1255.989114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1256.246138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1262.089170] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1265.460203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1269.232197] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1275.874182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1275.881896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.897372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1275.901250] Lustre: Skipped 1 previous similar message [ 1275.918067] Lustre: Skipped 1 previous similar message [ 1280.996587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1281.001147] Lustre: Skipped 1 previous similar message [ 1281.290089] Lustre: server umount lustre-OST0000 complete [ 1298.912068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1298.915574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.926232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1298.934253] Lustre: Skipped 2 previous similar messages [ 1301.474128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1301.476475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1306.594719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1306.603691] Lustre: Skipped 2 previous similar messages [ 1313.247113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1315.394244] Lustre: server umount lustre-MDT0000 complete [ 1317.856338] LustreError: 45197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1317.905099] LustreError: 45197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1319.355618] LustreError: 47788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792536 with bad export cookie 1240907430308747850 [ 1319.362283] LustreError: 47788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1319.362618] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1325.693817] Lustre: server umount lustre-MDT0001 complete [ 1335.926147] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1338.255334] Key type lgssc unregistered [ 1338.573447] LNet: 48926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1338.577421] LNetError: 48926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1338.590978] LNet: Removed LNI 192.168.201.139@tcp [ 1339.705588] Key type .llcrypt unregistered [ 1339.707381] Key type ._llcrypt unregistered [ 1364.576371] Key type ._llcrypt registered [ 1364.578519] Key type .llcrypt registered [ 1364.637922] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1365.668131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1365.682805] alg: No test for adler32 (adler32-zlib) [ 1366.911445] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 1367.211201] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1368.887229] Key type lgssc registered [ 1369.878451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1377.157743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1377.184239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1378.908713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1384.172981] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1389.025456] LustreError: 50349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1389.052897] LustreError: 50349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1393.228112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1393.675497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1398.804836] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1406.903774] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1412.678391] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1420.320516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1420.626698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1424.761864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1428.309179] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1437.115236] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1442.743731] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1446.621966] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1459.114214] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2621h04m35s [ 1460.664829] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1463.780916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.797919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1463.798701] Lustre: Skipped 1 previous similar message [ 1463.813853] Lustre: Skipped 1 previous similar message [ 1464.439208] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) [ 1464.446090] Lustre: Skipped 1 previous similar message [ 1465.502633] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) [ 1465.516617] Lustre: Skipped 3 previous similar messages [ 1469.128798] Lustre: server umount lustre-MDT0000 complete [ 1470.567297] LustreError: 50345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1473.680273] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1476.698868] LustreError: 50346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1476.714607] LustreError: 50346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1482.850303] LustreError: 53045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.886897] LustreError: 53045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1484.700952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1484.943865] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1485.413116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.553319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1486.466495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1492.344279] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1496.678476] Lustre: DEBUG MARKER: Wait for df (36692) ... [ 1499.036344] Lustre: DEBUG MARKER: done [ 1505.773182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1505.785255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1505.818216] Lustre: Skipped 1 previous similar message [ 1505.838112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1507.298766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.277533] Lustre: server umount lustre-OST0000 complete [ 1517.537673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.538413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1517.539651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1517.539656] Lustre: Skipped 2 previous similar messages [ 1517.544060] Lustre: Skipped 1 previous similar message [ 1528.806597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1528.827931] Lustre: Skipped 6 previous similar messages [ 1530.335323] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1530.676719] Lustre: server umount lustre-MDT0000 complete [ 1533.863915] LustreError: 50349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1533.879362] LustreError: 50349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1534.329902] LustreError: 50331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792751 with bad export cookie 12724784815273572557 [ 1534.331273] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1534.334887] LustreError: 50331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1534.611357] Lustre: server umount lustre-MDT0001 complete [ 1543.935885] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1546.343772] Key type lgssc unregistered [ 1546.763886] LNet: 55278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.782599] LNetError: 55278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1547.821176] LNet: Removed LNI 192.168.201.139@tcp [ 1548.887066] Key type .llcrypt unregistered [ 1548.892082] Key type ._llcrypt unregistered [ 1591.590635] Key type ._llcrypt registered [ 1591.592493] Key type .llcrypt registered [ 1591.691768] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1593.389968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1593.422650] alg: No test for adler32 (adler32-zlib) [ 1594.423888] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 1594.713624] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1596.391592] Key type lgssc registered [ 1597.951983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1605.380973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1605.405355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1606.621133] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1606.641350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1606.793174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1610.855805] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1621.690909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1621.756641] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1621.777420] Lustre: 57488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1621.988960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1626.326488] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1635.009325] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1641.793845] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1649.257845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1649.532281] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1649.787368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1655.963157] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1658.511266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1665.033255] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1673.461417] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2621h08m09s [ 1675.848669] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1684.475281] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1687.580427] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1687.641740] LustreError: 57498:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1708.170093] Lustre: lustre-MDT0000: Client 83475222-0518-46e3-95be-779cf7d26743 (at 192.168.201.39@tcp) reconnecting [ 1727.719257] LustreError: 57498:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1730.122284] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1733.296714] Lustre: DEBUG MARKER: done [ 1740.257362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1740.266163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1740.285893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1740.771553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1740.810724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1745.893287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1747.085470] Lustre: server umount lustre-OST0000 complete [ 1755.105335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1755.114302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.126593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1755.132312] Lustre: Skipped 1 previous similar message [ 1757.993862] Lustre: server umount lustre-MDT0000 complete [ 1762.284659] LustreError: 57498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1762.305028] LustreError: 57498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1763.026275] LustreError: 57482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792980 with bad export cookie 11874568560233228919 [ 1763.030222] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1763.047126] LustreError: 57482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1763.628124] Lustre: server umount lustre-MDT0001 complete [ 1777.868474] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1781.429256] Key type lgssc unregistered [ 1781.897993] LNet: 61825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.910789] LNetError: 61825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.948794] LNet: Removed LNI 192.168.201.139@tcp [ 1783.704586] Key type .llcrypt unregistered [ 1783.706614] Key type ._llcrypt unregistered [ 1833.634504] Key type ._llcrypt registered [ 1833.636849] Key type .llcrypt registered [ 1833.768528] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1834.435914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1834.469201] alg: No test for adler32 (adler32-zlib) [ 1835.652147] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 1835.946724] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1837.679123] Key type lgssc registered [ 1839.655737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1845.132400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1845.151500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.658507] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1846.697839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1847.037281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.702036] Lustre: Failing over lustre-MDT0000 [ 1849.092512] Lustre: server umount lustre-MDT0000 complete [ 1858.090694] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:11:14 (1787793074) [ 1864.959383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.432149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1870.445139] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1879.817337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.916819] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1879.946941] Lustre: 64638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1880.234472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1884.577211] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1893.461418] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1901.056429] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1908.697161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1908.999458] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1909.321055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1917.294067] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1919.208394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1925.655093] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1933.046531] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1941.986143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1942.003236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1942.043243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.057736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.060419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.098928] Lustre: Skipped 2 previous similar messages [ 1947.223122] Lustre: server umount lustre-MDT0000 complete [ 1951.733884] LustreError: 65372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793168 with bad export cookie 53530523616775949 [ 1951.747053] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1952.197703] Lustre: server umount lustre-MDT0001 complete [ 1956.548226] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1962.723832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1971.690251] Lustre: 63561:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787793172/real 1787793172] req@000000000223dd4d x1874636887337984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787793188 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1971.734810] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.747897] Lustre: Skipped 1 previous similar message [ 1972.768915] Lustre: 63561:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787793173/real 1787793173] req@00000000eae81dae x1874636887338112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787793189 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1973.260791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.799182] Lustre: 63562:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787793177/real 1787793177] req@00000000c9f882b1 x1874636887338240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787793193 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1976.869189] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbe2dba44d197f0 [ 1976.895668] Lustre: MGC192.168.201.139@tcp: Connection restored to 0@lo (at 0@lo) [ 1977.266330] LustreError: 66485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.293608] LustreError: 66485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1977.366074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1977.456973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1980.389606] LustreError: 66486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1981.422982] LustreError: 68405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1981.452180] LustreError: 68405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1982.506626] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1983.456289] Lustre: 63561:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787793183/real 1787793183] req@00000000c59e5341 x1874636887338624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787793199 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1983.477480] Lustre: 63561:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1986.531411] LustreError: 66485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1986.561500] LustreError: 66485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1987.551451] Lustre: 63560:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787793188/real 1787793188] req@00000000620a3ec6 x1874636887338752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787793204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1991.654584] LustreError: 68405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1991.682334] LustreError: 68405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1992.380389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.828376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1997.784708] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1998.314094] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2000.368459] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2005.739766] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2011.211926] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2018.273356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2018.286171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.304142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.173639] Lustre: server umount lustre-MDT0000 complete [ 2020.834427] LustreError: 66485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.857296] LustreError: 66485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2023.341770] LustreError: 65372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793240 with bad export cookie 53530523616778224 [ 2023.345393] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2023.348718] LustreError: 65372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2023.629223] Lustre: server umount lustre-MDT0001 complete [ 2030.140033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2040.424135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.056654] Lustre: 63563:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787793242/real 1787793242] req@000000009edd1e91 x1874636887365760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787793258 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2041.084315] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.103615] Lustre: Skipped 2 previous similar messages [ 2048.728206] LustreError: 66486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2048.834319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.898579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2053.881115] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2060.267969] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2063.857571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.500583] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2069.654717] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2079.612842] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2086.266561] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2094.672807] Lustre: server umount lustre-MDT0000 complete [ 2095.072653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2095.085931] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.087109] LustreError: 71089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2095.087118] LustreError: 71089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2095.153942] Lustre: Skipped 2 previous similar messages [ 2098.736778] LustreError: 64631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793315 with bad export cookie 53530523616779603 [ 2098.750895] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2098.757656] LustreError: 64631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2099.099147] Lustre: server umount lustre-MDT0001 complete [ 2106.211289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2110.807344] Lustre: DEBUG MARKER: files compared the same [ 2123.838063] Lustre: server umount lustre-OST0000 complete [ 2143.872935] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 2147.360274] Key type lgssc unregistered [ 2147.755910] LNet: 74369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2147.764257] LNetError: 74369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2147.782682] LNet: Removed LNI 192.168.201.139@tcp [ 2148.882299] Key type .llcrypt unregistered [ 2148.886048] Key type ._llcrypt unregistered [ 2175.901967] Key type ._llcrypt registered [ 2175.904491] Key type .llcrypt registered [ 2176.022143] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2178.017165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2178.127992] alg: No test for adler32 (adler32-zlib) [ 2179.354491] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 2179.753654] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 2181.543811] Key type lgssc registered [ 2183.170278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2192.994864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2193.028941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2194.907456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2200.761966] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug +malloc all [ 2204.129416] LustreError: 75789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.166484] LustreError: 75789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2209.248956] LustreError: 75790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.406761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2210.760571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2216.331763] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug +malloc all [ 2227.125574] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2236.745846] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2245.671735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2246.267801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2254.402644] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug +malloc all [ 2263.720446] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2264.717538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2275.316074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2275.329523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.349275] Lustre: Skipped 1 previous similar message [ 2275.356132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2275.375378] Lustre: Skipped 1 previous similar message [ 2280.416825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2281.219587] Lustre: server umount lustre-OST0000 complete [ 2285.543816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2285.546954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2285.559397] Lustre: Skipped 1 previous similar message [ 2285.571336] Lustre: Skipped 2 previous similar messages [ 2290.657477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2290.668345] Lustre: Skipped 1 previous similar message [ 2292.347302] Lustre: server umount lustre-MDT0000 complete [ 2295.782912] LustreError: 75790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.821848] LustreError: 75790:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2297.120687] LustreError: 75769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793514 with bad export cookie 5628514522045034757 [ 2297.121020] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2297.132558] LustreError: 75769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2297.416009] Lustre: server umount lustre-MDT0001 complete [ 2310.029421] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 2313.857957] Key type lgssc unregistered [ 2314.221203] LNet: 79329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2314.232258] LNetError: 79329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2314.251321] LNet: Removed LNI 192.168.201.139@tcp [ 2315.464558] Key type .llcrypt unregistered [ 2315.468610] Key type ._llcrypt unregistered [ 2344.107424] Key type ._llcrypt registered [ 2344.109103] Key type .llcrypt registered [ 2344.207532] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2346.246909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2346.310088] alg: No test for adler32 (adler32-zlib) [ 2347.353374] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 2347.841334] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 2349.687195] Key type lgssc registered [ 2351.537829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2363.681299] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2363.693172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2379.295577] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2384.416429] LustreError: 80721:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.139@tcp: failed processing log, type 4: rc = -110 [ 2411.232381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2418.578405] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2427.369339] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2436.145381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2436.265969] Lustre: *** cfs_fail_loc=706, val=0*** [ 2436.736706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2436.806286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2441.454065] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2442.210968] LustreError: 80747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.232750] LustreError: 80747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2445.284136] LustreError: 80748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2445.296089] LustreError: 80748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2450.406802] LustreError: 81808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.436600] LustreError: 81808:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2451.605369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.075257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2457.581385] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2465.967611] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2473.100688] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2477.542725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2477.550685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.557466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2477.570059] Lustre: Skipped 1 previous similar message [ 2477.585901] Lustre: Skipped 1 previous similar message [ 2481.884911] Lustre: server umount lustre-OST0000 complete [ 2487.781462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2487.782170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.783429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2487.799105] LustreError: Skipped 1 previous similar message [ 2487.852175] Lustre: Skipped 1 previous similar message [ 2492.839744] Lustre: server umount lustre-MDT0000 complete [ 2492.900378] LustreError: 81813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.929460] LustreError: 81813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2497.126256] LustreError: 80728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793714 with bad export cookie 11657290317482812635 [ 2497.133674] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2497.839615] Lustre: server umount lustre-MDT0001 complete [ 2509.311703] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 2512.481478] Key type lgssc unregistered [ 2512.880119] LNet: 84342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2512.890466] LNetError: 84342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2512.911081] LNet: Removed LNI 192.168.201.139@tcp [ 2513.873693] Key type .llcrypt unregistered [ 2513.875622] Key type ._llcrypt unregistered [ 2543.244404] Key type ._llcrypt registered [ 2543.246141] Key type .llcrypt registered [ 2543.381806] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2545.077246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2545.181907] alg: No test for adler32 (adler32-zlib) [ 2546.169486] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 2546.406518] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 2548.079193] Key type lgssc registered [ 2549.479421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2558.302601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2558.326839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.153570] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2574.679478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.549524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2580.207403] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2580.961394] LustreError: 86447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.990244] LustreError: 86447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2585.061917] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.089678] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2589.335478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2589.819047] LustreError: 86447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.841821] LustreError: 86447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2589.879146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2595.302749] LustreError: 86451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.326616] LustreError: 86451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2596.531205] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2600.424655] LustreError: 86446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.440794] LustreError: 86446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2605.351724] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2605.365977] Lustre: MGS: Client f8fa138d-4571-449e-9c2a-d4de3cd58e2f (at 0@lo) reconnecting [ 2605.395591] LustreError: 86461:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2605.542046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2605.550442] Lustre: Skipped 2 previous similar messages [ 2605.660787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.746292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2611.344530] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2626.016891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2626.027723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.029214] LustreError: Skipped 1 previous similar message [ 2626.032214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2626.046704] Lustre: Skipped 1 previous similar message [ 2631.139163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2631.144675] Lustre: Skipped 1 previous similar message [ 2632.015874] Lustre: server umount lustre-OST0000 complete [ 2636.257593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.268059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2636.281439] Lustre: Skipped 1 previous similar message [ 2641.379394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2641.385471] Lustre: Skipped 1 previous similar message [ 2642.216361] Lustre: server umount lustre-MDT0000 complete [ 2646.278891] Lustre: server umount lustre-MDT0001 complete [ 2651.177913] Lustre: server umount MGS complete [ 2664.233777] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:24:40 (1787793880) [ 2698.096498] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 2708.500727] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2749.481664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.243838] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2766.591037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.046416] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2767.131908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2767.248484] Lustre: lustre-MDT0001: new disk, initializing [ 2767.405329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2767.473277] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2767.479066] Lustre: 93639:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2767.485401] LustreError: 93639:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2772.971755] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2784.703396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2785.153849] Lustre: lustre-OST0000: new disk, initializing [ 2785.163848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2785.177484] Lustre: 94405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2785.280272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2792.345676] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2793.057732] Lustre: 94725:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000029348793 x1874637632872192/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787794026 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2802.422142] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2808.748053] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2808.770168] Lustre: MGS: Client 0b13fe71-9c20-49e2-8447-6f209ba0695b (at 0@lo) reconnecting [ 2808.817497] Lustre: 95452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2808.831874] LustreError: 93625:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2808.841143] LustreError: 93625:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2808.860663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2808.930415] Lustre: lustre-MDT0000: new disk, initializing [ 2809.110073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2809.142596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2809.163147] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2810.212373] Lustre: 95476:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000e0f61c75 x1874637632879104/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787794043 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2813.468199] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2815.344821] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2819.301446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2819.313513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2819.456460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2846.180179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2846.197833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.214929] Lustre: Skipped 1 previous similar message [ 2846.223699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2852.139072] Lustre: server umount lustre-OST0000 complete [ 2857.441230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2857.459030] Lustre: Skipped 2 previous similar messages [ 2861.869923] Lustre: server umount lustre-MDT0000 complete [ 2862.568795] LustreError: 93609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.593333] LustreError: 93609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2872.911759] Lustre: server umount lustre-MDT0001 complete [ 2877.325770] Lustre: server umount MGS complete [ 2889.455485] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:28:26 (1787794106) [ 2909.613348] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 2912.680040] Key type lgssc unregistered [ 2912.942707] LNet: 97979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2912.951319] LNetError: 97979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2912.971901] LNet: Removed LNI 192.168.201.139@tcp [ 2914.075964] Key type .llcrypt unregistered [ 2914.078941] Key type ._llcrypt unregistered [ 2936.327866] Key type ._llcrypt registered [ 2936.329796] Key type .llcrypt registered [ 2936.422186] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2937.636772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2937.694769] alg: No test for adler32 (adler32-zlib) [ 2938.811990] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 2939.145582] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 2940.887545] Key type lgssc registered [ 2942.460746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2951.105510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2951.119642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2952.617808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2955.874930] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2962.915569] LustreError: 99203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2962.953288] LustreError: 99203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2965.107086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.484054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2970.426509] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2979.260447] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2986.109160] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2993.967664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2994.442242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2997.556187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3001.222928] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3009.759862] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3018.209091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3018.218145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.243537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3021.283199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.289446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3023.810089] Lustre: server umount lustre-OST0000 complete [ 3031.520762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3031.532445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3031.536905] Lustre: Skipped 2 previous similar messages [ 3036.641201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3036.648560] Lustre: Skipped 1 previous similar message [ 3041.763127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3041.773743] Lustre: Skipped 1 previous similar message [ 3042.271301] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3042.495893] Lustre: server umount lustre-MDT0000 complete [ 3046.214574] LustreError: 99183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794263 with bad export cookie 2005859311285544036 [ 3046.226674] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3046.755671] Lustre: server umount lustre-MDT0001 complete [ 3058.381680] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 3060.876485] Key type lgssc unregistered [ 3061.124046] LNet: 102750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3061.128445] LNetError: 102750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3062.188120] LNet: Removed LNI 192.168.201.139@tcp [ 3063.281263] Key type .llcrypt unregistered [ 3063.283565] Key type ._llcrypt unregistered [ 3084.500786] Key type ._llcrypt registered [ 3084.505286] Key type .llcrypt registered [ 3084.668756] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3085.972524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3086.022542] alg: No test for adler32 (adler32-zlib) [ 3087.354444] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 3087.740257] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 3089.463313] Key type lgssc registered [ 3090.793962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3097.482378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3097.487787] LustreError: 103826:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3097.541731] LustreError: 103833:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3097.547474] LustreError: 103826:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4951 [ 3097.554270] LustreError: 103826:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3097.566382] LustreError: 103826:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3097.570528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3099.248322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3107.231065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.694606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3112.750859] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3119.813989] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing lsmod [ 3123.285940] LustreError: 105038:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3123.306357] LustreError: 105042:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3123.319338] LustreError: 105038:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4975 [ 3123.326856] LustreError: 105038:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3123.337929] LustreError: 105038:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3123.368866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3123.988572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3125.006388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3128.289405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3128.298527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.323641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3130.345687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3130.355733] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.360494] Lustre: Skipped 1 previous similar message [ 3130.392600] Lustre: Skipped 1 previous similar message [ 3132.455871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3132.473636] Lustre: Skipped 3 previous similar messages [ 3137.510780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3137.544586] Lustre: Skipped 2 previous similar messages [ 3142.111222] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3142.559681] Lustre: server umount lustre-MDT0000 complete [ 3142.625238] LustreError: 103870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.649046] LustreError: 103870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3147.116766] LustreError: 103852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794364 with bad export cookie 663812617182558226 [ 3147.122761] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3147.128807] LustreError: 103852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3147.748806] LustreError: 103866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3147.756814] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3147.784233] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3148.961042] LustreError: 104301:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000400944aa x1874638200135424/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3149.350448] Lustre: server umount lustre-MDT0001 complete [ 3154.605526] Lustre: server umount lustre-OST0000 complete [ 3164.086473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.473085] LustreError: 106108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3164.545318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3169.410820] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3169.762197] LustreError: 106109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3174.881513] LustreError: 106108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3179.876070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.360294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3185.316512] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3194.381143] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3202.308901] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3210.146269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3210.738050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3218.838853] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3227.260470] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3234.299661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3245.537468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3245.560541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.565841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3249.635643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3249.645798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3251.252558] Lustre: server umount lustre-OST0000 complete [ 3257.313245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3259.876277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.897251] Lustre: Skipped 1 previous similar message [ 3261.669862] Lustre: server umount lustre-MDT0000 complete [ 3264.998965] LustreError: 107104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3265.033947] LustreError: 107104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3265.620447] LustreError: 108716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794482 with bad export cookie 663812617182559157 [ 3265.623641] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3265.638771] LustreError: 108716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3266.100467] Lustre: server umount lustre-MDT0001 complete [ 3277.503534] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 3280.356870] Key type lgssc unregistered [ 3280.675165] LNet: 109672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3280.691728] LNetError: 109672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3280.717851] LNet: Removed LNI 192.168.201.139@tcp [ 3281.849521] Key type .llcrypt unregistered [ 3281.851541] Key type ._llcrypt unregistered [ 3307.513759] Key type ._llcrypt registered [ 3307.517432] Key type .llcrypt registered [ 3307.650306] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3309.025167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3309.115331] alg: No test for adler32 (adler32-zlib) [ 3310.100233] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 3310.367446] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 3312.047189] Key type lgssc registered [ 3313.294666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3321.941386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3321.952158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.502811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.097825] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3333.093871] LustreError: 111091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.114942] LustreError: 111091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3337.706661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3338.235772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3343.046678] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3351.813194] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3358.127911] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3364.465612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3364.879876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3369.014389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3371.035735] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3377.784240] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3386.086952] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3389.788631] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3394.529515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3394.535935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3394.545951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3398.625261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3398.628225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3398.655688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3399.927141] Lustre: server umount lustre-OST0000 complete [ 3403.752176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3403.757164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3403.767489] Lustre: Skipped 1 previous similar message [ 3403.773533] Lustre: Skipped 1 previous similar message [ 3408.865375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3408.868445] Lustre: Skipped 2 previous similar messages [ 3409.905366] Lustre: server umount lustre-MDT0000 complete [ 3413.229938] LustreError: 113624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794630 with bad export cookie 1948515134449814906 [ 3413.232749] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3413.238362] LustreError: 113624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3413.570684] Lustre: server umount lustre-MDT0001 complete [ 3424.152604] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 3426.987386] Key type lgssc unregistered [ 3427.274804] LNet: 114736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3427.283382] LNetError: 114736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3427.315856] LNet: Removed LNI 192.168.201.139@tcp [ 3428.243960] Key type .llcrypt unregistered [ 3428.246318] Key type ._llcrypt unregistered [ 3445.274323] Key type ._llcrypt registered [ 3445.276837] Key type .llcrypt registered [ 3445.386645] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3446.422135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3446.501457] alg: No test for adler32 (adler32-zlib) [ 3447.525624] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 3447.734357] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 3449.383165] Key type lgssc registered [ 3450.759603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3458.431701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3458.452854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3460.015979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3465.129698] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3469.289356] LustreError: 115959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3469.317303] LustreError: 115959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3474.404838] LustreError: 115958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3475.430907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.748936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3481.371662] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3490.381410] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3497.605370] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3506.252288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3506.626508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3513.831738] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3521.216903] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3529.164560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3541.472334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3541.479770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3541.508736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3541.994461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3542.001631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.017510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3544.547450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3544.553092] Lustre: Skipped 1 previous similar message [ 3547.500525] Lustre: server umount lustre-OST0000 complete [ 3552.745937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3552.757914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3552.792840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3554.787891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3557.759285] Lustre: server umount lustre-MDT0000 complete [ 3559.907822] LustreError: 115959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3559.926421] LustreError: 115959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3561.851710] LustreError: 115939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794778 with bad export cookie 9553345572288102470 [ 3561.860511] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3561.867896] LustreError: 115939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3562.262814] Lustre: server umount lustre-MDT0001 complete [ 3572.781429] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 3575.809393] Key type lgssc unregistered [ 3576.112302] LNet: 119516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3576.122494] LNetError: 119516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3576.142656] LNet: Removed LNI 192.168.201.139@tcp [ 3576.922459] Key type .llcrypt unregistered [ 3576.924621] Key type ._llcrypt unregistered [ 3598.549442] Key type ._llcrypt registered [ 3598.551573] Key type .llcrypt registered [ 3598.691534] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3599.845184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3599.988935] alg: No test for adler32 (adler32-zlib) [ 3600.988806] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 3601.251903] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 3602.951807] Key type lgssc registered [ 3603.978445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3611.729697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3611.745318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3613.261424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3618.526418] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3623.394668] LustreError: 120911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3623.408116] LustreError: 120911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3627.856739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3628.261772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3632.504458] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3640.977750] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3648.607682] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3656.646073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3656.831901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3658.897819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3664.404088] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3673.960633] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3681.836945] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3685.959985] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3685.965068] Lustre: Skipped 2 previous similar messages [ 3689.110816] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3689.198141] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3690.784892] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3690.794708] Lustre: Skipped 2 previous similar messages [ 3694.820268] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3694.826092] Lustre: Skipped 1 previous similar message [ 3700.859032] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.39@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3700.868918] Lustre: Skipped 2 previous similar messages [ 3701.317677] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.39@tcp 192.168.0.[2,10]@tcp [ 3711.884709] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.39/24@tcp in log lustre-MDT0000 [ 3711.891800] Lustre: Skipped 2 previous similar messages [ 3717.821072] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.39/24@tcp [ 3717.850573] Lustre: Skipped 1 previous similar message [ 3724.770214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3724.775758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3724.793290] Lustre: Skipped 1 previous similar message [ 3724.818156] Lustre: Skipped 1 previous similar message [ 3726.779987] Lustre: server umount lustre-OST0000 complete [ 3735.009857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3735.030764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3735.043216] Lustre: Skipped 1 previous similar message [ 3738.072540] Lustre: server umount lustre-MDT0000 complete [ 3740.129701] LustreError: 120906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3740.129701] LustreError: 120911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3741.863359] LustreError: 120893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794958 with bad export cookie 12258054793667648472 [ 3741.866105] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3741.873865] LustreError: 120893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3742.323913] Lustre: server umount lustre-MDT0001 complete [ 3755.142695] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 3758.104578] Key type lgssc unregistered [ 3758.416545] LNet: 125718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3758.422807] LNetError: 125718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3759.462888] LNet: Removed LNI 192.168.201.139@tcp [ 3760.326208] Key type .llcrypt unregistered [ 3760.332034] Key type ._llcrypt unregistered [ 3787.316186] Key type ._llcrypt registered [ 3787.317561] Key type .llcrypt registered [ 3787.408753] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3788.819596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3788.845988] alg: No test for adler32 (adler32-zlib) [ 3789.994257] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 3790.196293] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 3791.895196] Key type lgssc registered [ 3793.113760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3800.005402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3800.023133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.498390] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3801.539529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.550434] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.39/24@tcp [ 3805.868437] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3811.814889] LustreError: 127272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3811.826414] LustreError: 127272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3814.163479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3814.643466] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3814.692693] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.39/24@tcp [ 3814.754893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3819.193538] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3826.817938] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3832.684965] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3840.066665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3840.451415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3846.605385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3847.385608] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3857.390733] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3872.224828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3872.239566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3872.259879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3875.306293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3875.308153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3875.314358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3877.019465] Lustre: server umount lustre-OST0000 complete [ 3881.443400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3881.456429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3881.476475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3881.493212] Lustre: Skipped 1 previous similar message [ 3885.539770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3885.540244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3887.899762] Lustre: server umount lustre-MDT0000 complete [ 3890.660664] LustreError: 127272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3890.684366] LustreError: 127272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3892.952422] LustreError: 127254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795109 with bad export cookie 1249061337527493822 [ 3892.961169] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3892.967435] LustreError: 127254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3893.321046] Lustre: server umount lustre-MDT0001 complete [ 3905.753743] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 3908.701495] Key type lgssc unregistered [ 3909.047890] LNet: 130865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3909.053903] LNetError: 130865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3909.082392] LNet: Removed LNI 192.168.201.139@tcp [ 3910.282830] Key type .llcrypt unregistered [ 3910.286763] Key type ._llcrypt unregistered [ 3939.746434] Key type ._llcrypt registered [ 3939.749966] Key type .llcrypt registered [ 3939.817507] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 3955.154427] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3956.696408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3956.755520] alg: No test for adler32 (adler32-zlib) [ 3957.768417] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 3957.965393] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 3959.607212] Key type lgssc registered [ 3961.105827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3997.683590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3997.730836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3999.153447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3999.232328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3999.390084] Lustre: lustre-MDT0000: new disk, initializing [ 3999.467193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3999.489298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4004.816761] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4016.552356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4016.680631] Lustre: 134269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4016.758308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4016.764501] Lustre: Skipped 1 previous similar message [ 4016.864747] Lustre: lustre-MDT0001: new disk, initializing [ 4016.950206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4017.007415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4017.025597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4021.780782] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4033.098397] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4040.854373] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4050.765386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4051.220906] Lustre: lustre-OST0000: new disk, initializing [ 4051.223813] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4051.229408] Lustre: 136241:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4051.322780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4058.744768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4058.749774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4058.942221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4059.791467] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4069.700383] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4075.333610] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4075.532637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4080.730191] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4080.913765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4084.708820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4084.711674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4084.716603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4084.719906] Lustre: Skipped 1 previous similar message [ 4084.758103] LustreError: Skipped 1 previous similar message [ 4089.035961] Lustre: server umount lustre-OST0000 complete [ 4093.926768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4093.934808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4093.950851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.958068] Lustre: Skipped 1 previous similar message [ 4094.947465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4094.954857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4094.970883] Lustre: Skipped 1 previous similar message [ 4099.030131] Lustre: server umount lustre-MDT0000 complete [ 4102.954561] LustreError: 134260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795319 with bad export cookie 1738286966880083968 [ 4102.973484] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4103.266326] Lustre: server umount lustre-MDT0001 complete [ 4112.001213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4112.254946] LustreError: 138458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4112.261609] LustreError: 138458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4112.294419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4117.475630] LustreError: 138459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4118.117179] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4122.598848] LustreError: 138458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4127.715746] LustreError: 138459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4129.606507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4129.998357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4135.850544] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4145.340266] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4153.038875] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4161.227979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4161.699948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4168.904206] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4177.543512] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4183.218548] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4183.476680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4189.313346] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4189.618639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4200.057768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4200.399139] Lustre: lustre-OST0001: new disk, initializing [ 4200.404641] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4200.411548] Lustre: 141831:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4200.535644] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4207.519850] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4208.256222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4208.270420] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4208.330349] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4216.213573] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4216.426854] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4221.680263] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4222.007372] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4228.126602] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ad50854f800.ost_server_uuid 50 [ 4229.941391] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ad50854f800.ost_server_uuid in IDLE state after 0 sec [ 4239.338479] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4239.348612] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4239.361167] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4239.367101] Lustre: Skipped 1 previous similar message [ 4244.471703] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4244.478077] Lustre: Skipped 1 previous similar message [ 4245.250860] Lustre: server umount lustre-OST0001 complete [ 4249.576050] LustreError: 140283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4249.615856] LustreError: 140283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4252.133072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4252.138152] LustreError: Skipped 1 previous similar message [ 4252.141988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.173880] Lustre: Skipped 1 previous similar message [ 4254.179331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4254.184762] Lustre: Skipped 1 previous similar message [ 4256.443995] Lustre: server umount lustre-OST0000 complete [ 4263.392476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4263.400682] LustreError: Skipped 1 previous similar message [ 4263.407989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.428585] Lustre: Skipped 1 previous similar message [ 4266.915634] Lustre: server umount lustre-MDT0000 complete [ 4270.050107] LustreError: 138454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4270.079628] LustreError: 138454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4271.054260] LustreError: 138439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795488 with bad export cookie 1738286966880084850 [ 4271.056625] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4271.061860] LustreError: 138439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4271.538352] Lustre: server umount lustre-MDT0001 complete [ 4291.300473] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 4294.323945] Key type lgssc unregistered [ 4294.536830] LNet: 144842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4294.556671] LNetError: 144842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4294.585726] LNet: Removed LNI 192.168.201.139@tcp [ 4295.464241] Key type .llcrypt unregistered [ 4295.467440] Key type ._llcrypt unregistered [ 4337.360915] Key type ._llcrypt registered [ 4337.363542] Key type .llcrypt registered [ 4337.491032] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 4353.474267] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 4354.542894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4354.587327] alg: No test for adler32 (adler32-zlib) [ 4355.782308] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 4356.010044] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 4357.687595] Key type lgssc registered [ 4358.767306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4397.647457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4397.685048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4398.995903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4399.051755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4399.131737] Lustre: lustre-MDT0000: new disk, initializing [ 4399.212828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4399.235837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4404.332850] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4415.254250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4415.416647] Lustre: 148996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4415.525205] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4415.531162] Lustre: Skipped 1 previous similar message [ 4415.601244] Lustre: lustre-MDT0001: new disk, initializing [ 4415.695494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4415.736683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4415.766355] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4420.714202] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4430.581887] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4436.343717] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4443.848258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4444.033330] Lustre: lustre-OST0000: new disk, initializing [ 4444.035422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4444.038828] Lustre: 150968:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4444.133797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4451.547260] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4454.474303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4454.494279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4454.604981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4461.904972] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4467.467562] Lustre: Failing over lustre-OST0000 [ 4467.689519] Lustre: server umount lustre-OST0000 complete [ 4484.585704] Lustre: 147223:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795685/real 1787795685] req@000000009ab0d4f9 x1874639529740800/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787795701 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4484.586345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4484.614833] Lustre: 147223:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4486.305217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4486.674129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4486.698209] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4487.648137] Lustre: 147220:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795687/real 1787795687] req@000000007d373c78 x1874639529741056/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787795703 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4489.695870] Lustre: 147221:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795690/real 1787795690] req@000000001734ef1b x1874639529742080/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787795706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4489.726217] Lustre: 147221:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4492.573167] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4494.819117] Lustre: 147220:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795695/real 1787795695] req@0000000068c85cd2 x1874639529743232/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787795711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4494.837586] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4494.846151] Lustre: 147220:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4496.292619] Lustre: Failing over lustre-MDT0000 [ 4496.307887] LustreError: 147219:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4496.544384] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4496.565521] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4496.732459] Lustre: server umount lustre-MDT0000 complete [ 4497.888637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4497.897583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4497.909640] Lustre: Skipped 1 previous similar message [ 4499.681078] Lustre: 147222:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795700/real 1787795700] req@0000000064cac68d x1874639529744384/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787795716 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4499.938957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4499.952559] Lustre: Skipped 1 previous similar message [ 4501.595283] LustreError: 150975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4501.630358] LustreError: 150975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4505.056647] LustreError: 149005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4505.086669] LustreError: 149005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4506.721816] LustreError: 150975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4510.179086] LustreError: 149820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4510.197882] LustreError: 149820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4515.039218] Lustre: 147220:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795716/real 1787795716] req@00000000c105d485 x1874639529753728/t0(0) o400->MGC192.168.201.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787795732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4515.053163] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4515.066028] LustreError: 149009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4515.085519] LustreError: 149009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4515.610150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4525.537501] LustreError: 147219:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000906ba83b x1874639529758336/t0(0) o250->MGC192.168.201.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4525.574452] LustreError: 149009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4525.611754] LustreError: 149009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4525.854437] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4525.969959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4526.021895] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4527.197214] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4531.211223] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4531.247646] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4531.325450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4531.765525] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4541.415761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4541.443885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4541.450261] Lustre: Skipped 1 previous similar message [ 4546.458744] Lustre: server umount lustre-OST0000 complete [ 4551.649668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4551.654507] Lustre: Skipped 1 previous similar message [ 4551.659660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4551.659840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4551.673865] Lustre: Skipped 1 previous similar message [ 4556.781646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4556.795775] Lustre: Skipped 1 previous similar message [ 4557.043864] Lustre: server umount lustre-MDT0000 complete [ 4561.195330] LustreError: 148991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795778 with bad export cookie 5956500917245846464 [ 4561.198174] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4561.205903] LustreError: 148991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4561.539361] Lustre: server umount lustre-MDT0001 complete [ 4573.186929] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 4575.557775] Key type lgssc unregistered [ 4575.868174] LNet: 154625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4575.877574] LNetError: 154625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4575.892519] LNet: Removed LNI 192.168.201.139@tcp [ 4576.690274] Key type .llcrypt unregistered [ 4576.692108] Key type ._llcrypt unregistered [ 4615.781311] Key type ._llcrypt registered [ 4615.785045] Key type .llcrypt registered [ 4615.925729] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 4617.948735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4618.017129] alg: No test for adler32 (adler32-zlib) [ 4619.295679] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 4619.596293] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 4621.263237] Key type lgssc registered [ 4622.888833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4633.694725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4633.715687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4635.689079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4640.625507] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4645.856795] LustreError: 156146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4645.869918] LustreError: 156146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4650.976404] LustreError: 156147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.078071] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4656.100052] LustreError: 156146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4656.452858] LustreError: 156175:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4656.461802] LustreError: 156175:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4656.469869] LustreError: 156175:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4656.908147] Lustre: server umount lustre-MDT0000 complete [ 4673.395837] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 4683.018896] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 4725.115572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4725.388270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4725.422240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4725.521156] Lustre: lustre-MDT0000: new disk, initializing [ 4725.616131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4725.645972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4730.317614] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4741.897967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4742.036341] Lustre: 159480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4742.067407] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4742.069203] Lustre: Skipped 1 previous similar message [ 4742.119436] Lustre: lustre-MDT0001: new disk, initializing [ 4742.220273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4742.291042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4742.300076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4747.366966] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4757.964753] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4763.938938] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4771.068026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4771.566109] Lustre: lustre-OST0000: new disk, initializing [ 4771.573686] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4771.586513] Lustre: 161449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4771.798566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.592477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4778.605716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4778.766556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4778.830519] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4789.987588] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4794.810917] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4796.571449] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:00:13 (1787796013) [ 5049.311758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5049.321253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5049.339305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5050.338240] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5050.339785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5050.363283] Lustre: Skipped 2 previous similar messages [ 5055.456588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5055.473684] Lustre: Skipped 2 previous similar messages [ 5060.577610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5060.588973] Lustre: Skipped 2 previous similar messages [ 5061.089825] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5061.450697] Lustre: server umount lustre-MDT0000 complete [ 5065.610916] LustreError: 159471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796282 with bad export cookie 10496056077299543161 [ 5065.613186] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5065.629432] LustreError: 159471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5065.699218] LustreError: 159487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5065.699562] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5065.716496] Lustre: Skipped 1 previous similar message [ 5065.718680] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5070.815632] LustreError: 163244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5072.104766] Lustre: server umount lustre-MDT0001 complete [ 5082.828807] Lustre: server umount lustre-OST0000 complete [ 5092.375435] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 5102.463124] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 5141.522215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5141.848911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5141.883067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5141.994321] Lustre: lustre-MDT0000: new disk, initializing [ 5142.198965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5142.255223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5148.012517] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5160.276398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5160.466295] Lustre: 166127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5160.578730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5160.588670] Lustre: Skipped 1 previous similar message [ 5160.725699] Lustre: lustre-MDT0001: new disk, initializing [ 5160.894508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5160.948272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5160.956973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5165.413988] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5176.022187] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5182.198572] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5189.070780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5189.507519] Lustre: lustre-OST0000: new disk, initializing [ 5189.513861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5189.522944] Lustre: 168096:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5189.647187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5197.361831] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5198.957988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5198.963599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5199.129418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5207.346251] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5212.299077] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5212.649816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5217.996338] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5218.243914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5224.929724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5224.939126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5224.941330] LustreError: Skipped 1 previous similar message [ 5224.945313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5224.950150] Lustre: Skipped 1 previous similar message [ 5224.982364] Lustre: Skipped 2 previous similar messages [ 5226.627810] Lustre: server umount lustre-OST0000 complete [ 5232.608838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5232.620928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5237.257566] Lustre: server umount lustre-MDT0000 complete [ 5240.295190] LustreError: 166133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5240.341750] LustreError: 166133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5242.312767] LustreError: 166119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796459 with bad export cookie 10496056077299701984 [ 5242.322858] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5242.327422] LustreError: 166119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5242.711940] Lustre: server umount lustre-MDT0001 complete [ 5252.254352] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:07:48 (1787796468) [ 5270.871072] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 5282.462540] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 5326.772199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5327.001892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5327.071991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5327.195122] Lustre: lustre-MDT0000: new disk, initializing [ 5327.326871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5327.371181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5332.889872] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5344.546217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5344.674336] Lustre: 172857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5344.900898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5349.989436] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5361.728995] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5368.780705] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5376.304349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5376.563392] Lustre: lustre-OST0000: new disk, initializing [ 5376.566802] Lustre: Skipped 1 previous similar message [ 5376.570631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5376.577921] Lustre: Skipped 2 previous similar messages [ 5376.583305] Lustre: 174835:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5382.977964] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5383.255927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5383.277537] Lustre: Skipped 1 previous similar message [ 5383.286281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5383.484518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5393.700163] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5404.132801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5404.150428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5404.163280] Lustre: Skipped 1 previous similar message [ 5404.164681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5404.164686] Lustre: Skipped 3 previous similar messages [ 5404.198831] LustreError: Skipped 1 previous similar message [ 5406.586567] Lustre: server umount lustre-OST0000 complete [ 5411.300443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5416.784447] Lustre: server umount lustre-MDT0000 complete [ 5419.502982] LustreError: 172868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5419.522226] LustreError: 172868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5421.186556] LustreError: 172848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796638 with bad export cookie 10496056077299702873 [ 5421.190487] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5421.211884] LustreError: 172848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5421.566506] Lustre: server umount lustre-MDT0001 complete [ 5431.591605] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:10:48 (1787796648) [ 5450.158644] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 5461.799739] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 5505.243043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5505.547876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5505.589314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5505.647424] Lustre: lustre-MDT0000: new disk, initializing [ 5505.716864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5505.721831] Lustre: Skipped 2 previous similar messages [ 5505.732685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5510.091158] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5520.792659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5520.903868] Lustre: 179160:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5521.082097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5525.654161] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5534.991831] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5540.599264] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5546.617491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5546.970573] Lustre: 181136:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5553.734621] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5556.259771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5556.368577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5563.362899] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5570.929846] Lustre: server umount lustre-OST0000 complete [ 5576.674253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5576.709334] Lustre: Skipped 3 previous similar messages [ 5576.720551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5576.731519] Lustre: Skipped 5 previous similar messages [ 5585.011547] Lustre: server umount lustre-MDT0000 complete [ 5586.913093] LustreError: 179174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5586.925670] LustreError: 179174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5587.935250] Lustre: 155457:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787796788/real 1787796788] req@000000008a51a540 x1874639807346944/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787796804 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5587.976778] Lustre: 155457:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5589.622770] LustreError: 179961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796806 with bad export cookie 10496056077299703958 [ 5589.624610] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5589.636745] LustreError: 179961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5590.013497] Lustre: server umount lustre-MDT0001 complete [ 5603.070670] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 5608.108901] Key type lgssc unregistered [ 5608.419140] LNet: 182996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5608.428479] LNetError: 182996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5608.445091] LNet: Removed LNI 192.168.201.139@tcp [ 5609.494507] Key type .llcrypt unregistered [ 5609.495840] Key type ._llcrypt unregistered [ 5634.097617] Key type ._llcrypt registered [ 5634.099220] Key type .llcrypt registered [ 5634.258298] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 5635.778105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5635.817332] alg: No test for adler32 (adler32-zlib) [ 5636.930586] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 5637.204238] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 5639.023200] Key type lgssc registered [ 5640.879856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5648.766403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5648.780531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5650.332817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5654.975647] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5660.644673] LustreError: 184414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5660.658599] LustreError: 184414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5663.195584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5663.556915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5668.338945] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5675.213112] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5680.367768] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5686.709647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5687.249522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5692.818555] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5695.481415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5699.793353] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5721.081309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5721.098379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5721.147153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5725.565248] Lustre: server umount lustre-OST0000 complete [ 5730.273129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5730.285416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5730.307600] Lustre: Skipped 1 previous similar message [ 5730.321954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5730.331380] Lustre: Skipped 1 previous similar message [ 5731.298877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5731.323219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5731.333163] Lustre: Skipped 1 previous similar message [ 5735.914602] Lustre: server umount lustre-MDT0000 complete [ 5736.419086] LustreError: 184414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5736.433924] LustreError: 184414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5739.044072] LustreError: 184396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796956 with bad export cookie 13340909226832419930 [ 5739.066326] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5739.397636] Lustre: server umount lustre-MDT0001 complete [ 5751.012475] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 5754.117301] Key type lgssc unregistered [ 5754.410851] LNet: 187972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5754.416281] LNetError: 187972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5754.427295] LNet: Removed LNI 192.168.201.139@tcp [ 5755.271099] Key type .llcrypt unregistered [ 5755.273826] Key type ._llcrypt unregistered [ 5779.121147] Key type ._llcrypt registered [ 5779.123557] Key type .llcrypt registered [ 5779.257136] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 5780.281275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5780.318479] alg: No test for adler32 (adler32-zlib) [ 5781.428570] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 5781.646761] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 5783.311242] Key type lgssc registered [ 5784.481391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5791.240686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5791.251270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5792.726831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5796.923721] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5802.979713] LustreError: 189388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5802.991778] LustreError: 189388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5805.936299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5806.473194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5810.819289] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5817.414515] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5822.034275] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5828.167640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5828.550280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5835.152268] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5837.754665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5843.324715] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5850.233036] Lustre: lustre-OST0000: Not available for connect from 192.168.201.39@tcp (stopping) [ 5852.129230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5852.136184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5852.143686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5853.153792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5853.163950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5853.167371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5853.192054] Lustre: Skipped 1 previous similar message [ 5855.329670] Lustre: lustre-OST0000: Not available for connect from 192.168.201.39@tcp (stopping) [ 5855.460933] Lustre: server umount lustre-OST0000 complete [ 5863.943710] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9ad503375800.ost_server_uuid 50 [ 5865.931048] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5883.366907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5883.390499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5883.430853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5885.882283] Lustre: server umount lustre-MDT0000 complete [ 5888.996109] LustreError: 189385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5889.033095] LustreError: 189385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5891.733387] LustreError: 190124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797108 with bad export cookie 4101574705235737770 [ 5891.751378] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5891.754040] LustreError: 190124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5892.104152] Lustre: server umount lustre-MDT0001 complete [ 5907.962873] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:18:43 (1787797123) [ 5917.362426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5917.848906] LustreError: 193305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5918.022768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5923.303092] LustreError: 193306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5924.674886] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5927.393515] LustreError: 193305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5933.906807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5934.291901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5939.464140] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5948.223638] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5955.281101] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5962.801534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5963.344377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5970.170656] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5979.440241] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5986.683488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5986.914184] Lustre: lustre-OST0001: new disk, initializing [ 5986.926985] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5986.938435] Lustre: 196152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5987.111080] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5988.145311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5993.772895] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5995.622321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5995.628487] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5995.743065] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6003.643064] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6006.244418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.247456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6006.251613] Lustre: Skipped 2 previous similar messages [ 6006.262502] Lustre: Skipped 2 previous similar messages [ 6012.372581] Lustre: server umount lustre-OST0000 complete [ 6014.564424] LustreError: 195131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6014.587310] LustreError: 195131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6019.104626] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6019.359969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6023.382067] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6023.705965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6031.861262] LustreError: 195128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6031.895142] LustreError: 195128:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6042.095923] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6042.098027] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6042.107288] LustreError: Skipped 1 previous similar message [ 6042.108485] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6042.108491] Lustre: Skipped 3 previous similar messages [ 6042.155031] Lustre: Skipped 1 previous similar message [ 6045.740843] Lustre: server umount lustre-OST0001 complete [ 6052.326790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6052.339413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6052.344262] Lustre: Skipped 1 previous similar message [ 6057.059224] Lustre: server umount lustre-MDT0000 complete [ 6060.603900] LustreError: 193286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797277 with bad export cookie 4101574705235738841 [ 6060.613780] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6060.614793] LustreError: 193286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6060.864893] Lustre: server umount lustre-MDT0001 complete [ 6085.546866] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:21:41 (1787797301) [ 6093.015992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6093.583512] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6093.617305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6093.939184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6100.072351] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6109.495216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6109.620857] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6109.662466] Lustre: 199588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6115.133197] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6122.410749] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6128.136605] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6135.810900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6136.039224] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6141.628687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6144.604733] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6152.847691] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6161.377152] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6161.486629] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6161.903401] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6161.913392] Lustre: Skipped 2 previous similar messages [ 6170.187981] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6178.605498] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6182.880671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6182.884903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6182.890295] LustreError: Skipped 1 previous similar message [ 6182.892145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6182.918481] Lustre: Skipped 1 previous similar message [ 6182.941132] Lustre: Skipped 4 previous similar messages [ 6187.723072] Lustre: server umount lustre-OST0000 complete [ 6189.660565] LustreError: 201438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6189.683050] LustreError: 201438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6205.922867] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6208.918434] Lustre: server umount lustre-OST0001 complete [ 6216.686490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6216.717838] Lustre: Skipped 3 previous similar messages [ 6218.524160] Lustre: server umount lustre-MDT0000 complete [ 6222.183058] LustreError: 199579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797439 with bad export cookie 4101574705235740248 [ 6222.184068] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6222.189022] LustreError: 199579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6222.456398] Lustre: server umount lustre-MDT0001 complete [ 6248.365511] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:24:24 (1787797464) [ 6267.706339] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 6276.829554] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 6313.813475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6314.183464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6314.214145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6314.296658] Lustre: lustre-MDT0000: new disk, initializing [ 6314.387712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6314.418582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6319.290245] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6330.781533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6330.937123] Lustre: 207686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6331.031817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6331.041348] Lustre: Skipped 1 previous similar message [ 6331.203287] Lustre: lustre-MDT0001: new disk, initializing [ 6331.435480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6331.447519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6336.328904] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6346.130658] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6352.628567] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6360.546540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6360.960464] Lustre: lustre-OST0000: new disk, initializing [ 6360.964888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6360.973393] Lustre: 209653:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6369.177991] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6369.914363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6369.928920] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6370.019421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6381.134316] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6387.763349] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6388.077507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6394.634902] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6394.968395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6401.026355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6401.030802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6401.049246] LustreError: Skipped 3 previous similar messages [ 6401.053518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6401.062984] Lustre: Skipped 1 previous similar message [ 6401.084803] Lustre: Skipped 10 previous similar messages [ 6403.605706] Lustre: server umount lustre-OST0000 complete [ 6413.902080] Lustre: server umount lustre-MDT0000 complete [ 6414.306101] LustreError: 207697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6414.327750] LustreError: 207697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6418.159516] LustreError: 207677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797635 with bad export cookie 4101574705235741802 [ 6418.161666] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6418.175787] LustreError: 207677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6418.633566] Lustre: server umount lustre-MDT0001 complete [ 6428.192977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6434.111069] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6443.139050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6448.831981] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6456.166247] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6462.186834] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6469.633594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6477.460146] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6486.493828] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6491.783274] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6492.009767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6497.373754] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6497.657680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6503.913183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6503.926565] LustreError: Skipped 1 previous similar message [ 6506.951952] Lustre: server umount lustre-OST0000 complete [ 6516.476869] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6523.389509] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6553.122493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6561.077429] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6568.944256] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6569.899226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6569.918193] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6569.919662] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6569.974621] Lustre: Skipped 1 previous similar message [ 6570.729398] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6574.879778] Lustre: server umount lustre-OST0000 complete [ 6580.195116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6580.195953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6580.210706] Lustre: Skipped 5 previous similar messages [ 6580.225288] Lustre: Skipped 6 previous similar messages [ 6589.503238] Lustre: server umount lustre-MDT0000 complete [ 6590.435194] LustreError: 214088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6590.460267] LustreError: 214088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6591.456660] Lustre: 188698:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787797792/real 1787797792] req@00000000d85ba1f7 x1874641025233536/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787797808 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6591.499025] Lustre: 188698:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6594.118229] LustreError: 211851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797811 with bad export cookie 4101574705235742691 [ 6594.123035] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6594.130228] LustreError: 211851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6594.453399] Lustre: server umount lustre-MDT0001 complete [ 6602.996475] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 22:30:19 (1787797819) [ 6609.090553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6609.530297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6609.536553] Lustre: Skipped 6 previous similar messages [ 6614.380537] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6623.403442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6628.767448] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6637.124660] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6643.763381] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6650.959729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6657.355425] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6664.787641] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6670.513497] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6670.774147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6676.170993] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6676.405491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6683.510197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6683.763298] Lustre: lustre-OST0001: new disk, initializing [ 6683.767218] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6683.773836] Lustre: 221211:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6690.984761] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6691.424618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6691.436774] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6691.559779] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6701.050547] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6705.914566] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6706.203330] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6710.694489] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6710.978361] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6717.410257] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6717.428421] LustreError: Skipped 1 previous similar message [ 6719.325822] Lustre: server umount lustre-OST0001 complete [ 6725.556866] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6725.769534] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6730.386482] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6730.602411] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6760.050823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6765.549486] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6765.562740] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6765.563112] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6765.578504] Lustre: Skipped 1 previous similar message [ 6766.906440] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6775.134738] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6777.003425] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6785.939786] Lustre: server umount lustre-OST0001 complete [ 6818.272907] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6818.424152] Lustre: server umount lustre-OST0000 complete [ 6828.606391] Lustre: server umount lustre-MDT0000 complete [ 6832.491913] LustreError: 217823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798049 with bad export cookie 4101574705235743657 [ 6832.498048] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6832.504302] LustreError: 217823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6838.986953] Lustre: server umount lustre-MDT0001 complete [ 6862.214496] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 22:34:38 (1787798078) [ 6868.298494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6868.472929] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6868.493573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6873.443378] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6882.168268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6882.246250] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6882.274553] Lustre: 226456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6887.325422] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6896.022988] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6901.580439] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6909.474566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6909.846451] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6916.642787] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6924.841247] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6934.388126] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6934.523565] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6941.450900] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6948.684425] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6950.001557] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6954.162291] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6954.401435] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6959.089476] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6959.410605] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6964.763657] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ad511e67000.ost_server_uuid 50 [ 6966.119358] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ad511e67000.ost_server_uuid in IDLE state after 0 sec [ 6966.975712] Lustre: Permanently deactivating lustre-OST0001 [ 6966.996631] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6971.590183] Lustre: Permanently reactivating lustre-OST0001 [ 6971.624168] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6971.630871] Lustre: Skipped 2 previous similar messages [ 6976.996519] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6977.011746] LustreError: Skipped 3 previous similar messages [ 6977.020681] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6977.037837] Lustre: Skipped 9 previous similar messages [ 6977.048106] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6977.050366] Lustre: Skipped 15 previous similar messages [ 6990.303167] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6990.579939] Lustre: server umount lustre-OST0001 complete [ 6993.894039] LustreError: 228301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6993.932113] LustreError: 228301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 44 previous similar messages [ 7009.247159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7009.459968] Lustre: server umount lustre-OST0000 complete [ 7028.191237] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7028.427430] Lustre: server umount lustre-MDT0000 complete [ 7032.132884] LustreError: 226449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798249 with bad export cookie 4101574705235744945 [ 7032.134507] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7032.163169] LustreError: 226449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7032.523590] Lustre: server umount lustre-MDT0001 complete [ 7057.868038] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 22:37:54 (1787798274) [ 7065.772899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7065.991879] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7066.034763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7066.047736] Lustre: Skipped 2 previous similar messages [ 7070.469556] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7078.530852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7078.608140] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7078.633558] Lustre: 233149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7083.614604] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7091.738346] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7097.147091] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7104.188468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7104.565036] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7104.641501] Lustre: Permanently deactivating lustre-OST0000 [ 7104.692189] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7111.759559] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7111.968355] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7119.555322] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7126.243730] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7126.309429] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7126.496167] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7126.498489] Lustre: Skipped 11 previous similar messages [ 7133.435686] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7134.357500] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7141.363902] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7142.950290] Lustre: Permanently reactivating lustre-OST0000 [ 7142.971335] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7142.978271] Lustre: Skipped 2 previous similar messages [ 7151.727713] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7151.733215] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 40 seconds. I think it's dead, and I am evicting it. exp 00000000db6d1357, cur 1787798368 deadline 1787798358 last 1787798328 [ 7151.744093] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7151.773869] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7151.870617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7157.764567] Lustre: server umount lustre-OST0001 complete [ 7167.857556] Lustre: server umount lustre-OST0000 complete [ 7178.666918] Lustre: server umount lustre-MDT0000 complete [ 7182.242485] LustreError: 233889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798399 with bad export cookie 4101574705235746583 [ 7182.252480] LustreError: 233889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7182.255480] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7182.541441] Lustre: server umount lustre-MDT0001 complete [ 7192.836694] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7196.790972] Key type lgssc unregistered [ 7197.137568] LNet: 238031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7197.141552] LNetError: 238031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7198.185228] LNet: Removed LNI 192.168.201.139@tcp [ 7199.272486] Key type .llcrypt unregistered [ 7199.277073] Key type ._llcrypt unregistered [ 7225.067299] Key type ._llcrypt registered [ 7225.068229] Key type .llcrypt registered [ 7225.170573] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7226.147437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7226.361084] alg: No test for adler32 (adler32-zlib) [ 7227.507319] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 7227.785371] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 7229.498216] Key type lgssc registered [ 7230.655938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7242.149686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7242.161224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7243.520556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7248.176753] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7254.018750] LustreError: 239507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7254.045922] LustreError: 239507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7256.679389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7256.798298] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7256.844109] Lustre: 239497:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7256.858071] Lustre: Permanently deactivating lustre-MDT0001 [ 7256.889829] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7257.184850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7259.251595] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7259.258829] LustreError: 239538:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7259.274765] LustreError: 239538:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7259.290515] LustreError: 239538:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7262.453251] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7271.707502] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7277.618985] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7284.735498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7285.150599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7289.249654] LustreError: 241347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7289.292026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7292.090867] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7294.436198] LustreError: 241348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7298.950314] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7299.563309] LustreError: 241347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7304.914622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7305.100897] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7310.336876] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7310.357537] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7310.572564] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7317.749560] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7322.115889] Lustre: Permanently reactivating lustre-MDT0001 [ 7322.168939] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7322.176417] Lustre: Skipped 2 previous similar messages [ 7325.739491] Lustre: lustre-MDT0001: Client 285fd21e-bc3f-4910-9768-258254fd1d88 (at 192.168.201.39@tcp) reconnecting [ 7333.881929] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7336.475587] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7343.865856] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7344.167892] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7349.741851] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7350.034607] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7352.660888] Lustre: Permanently deactivating lustre-MDT0001 [ 7352.692872] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7352.708055] Lustre: Skipped 2 previous similar messages [ 7360.654886] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7369.165598] Lustre: server umount lustre-MDT0000 complete [ 7369.700621] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7369.702087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7369.705372] LustreError: 239502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7369.705384] LustreError: 239502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7369.782706] Lustre: Skipped 3 previous similar messages [ 7372.753483] LustreError: 243989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798589 with bad export cookie 14324964684328655239 [ 7372.766463] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7372.777180] LustreError: 243989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7373.090868] Lustre: server umount lustre-MDT0001 complete [ 7376.968691] Lustre: server umount lustre-OST0000 complete [ 7388.571384] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 22:43:25 (1787798605) [ 7398.824064] Lustre: server umount lustre-OST0001 complete [ 7405.331476] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 7415.547209] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7455.684504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7456.035933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7456.052172] Lustre: Skipped 2 previous similar messages [ 7456.075023] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7456.188511] Lustre: lustre-MDT0000: new disk, initializing [ 7456.323410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7456.342833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7460.753239] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7471.133360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7471.250835] Lustre: 247763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7471.288308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7471.294808] Lustre: Skipped 1 previous similar message [ 7471.376315] Lustre: lustre-MDT0001: new disk, initializing [ 7471.482841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7471.508587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7471.523492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7475.849547] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7485.240672] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7490.131266] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7496.249032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7496.638842] Lustre: lustre-OST0000: new disk, initializing [ 7496.642375] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7496.648408] Lustre: 249733:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7496.741169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7502.505732] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7506.989067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7506.995285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7507.112191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7511.936262] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7513.551215] LustreError: 249741:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7521.496075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7521.700978] Lustre: lustre-OST0001: new disk, initializing [ 7521.704297] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7521.709250] Lustre: 250849:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7523.559157] LustreError: 249741:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7523.579594] LustreError: 247773:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7528.402876] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7530.149341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7530.171175] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7530.247825] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7533.672083] LustreError: 247773:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7538.521902] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7540.038959] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7544.453388] Lustre: lustre-OST0001: Not available for connect from 192.168.201.39@tcp (stopping) [ 7545.842577] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.851995] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7545.852623] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7545.854568] Lustre: Skipped 1 previous similar message [ 7549.534863] Lustre: lustre-OST0001: Not available for connect from 192.168.201.39@tcp (stopping) [ 7554.660286] Lustre: lustre-OST0001: Not available for connect from 192.168.201.39@tcp (stopping) [ 7554.678354] Lustre: Skipped 2 previous similar messages [ 7556.069070] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7556.272944] Lustre: server umount lustre-OST0001 complete [ 7559.772929] LustreError: 249715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7559.782763] LustreError: 249715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7573.985054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7573.987402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7573.989655] LustreError: Skipped 2 previous similar messages [ 7573.996741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7574.011880] Lustre: Skipped 2 previous similar messages [ 7574.053726] Lustre: Skipped 3 previous similar messages [ 7576.546618] LustreError: 249717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7576.572701] LustreError: 249717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7578.679995] Lustre: server umount lustre-OST0000 complete [ 7584.226472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7584.233531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7584.252468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7584.256088] Lustre: Skipped 2 previous similar messages [ 7589.275501] Lustre: server umount lustre-MDT0000 complete [ 7592.924160] LustreError: 247757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798809 with bad export cookie 14324964684328657836 [ 7592.931089] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7592.936924] LustreError: 247757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7607.263148] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7607.906436] Lustre: server umount lustre-MDT0001 complete [ 7617.970135] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7620.590890] Key type lgssc unregistered [ 7620.832774] LNet: 252885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7620.849076] LNetError: 252885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7620.897426] LNet: Removed LNI 192.168.201.139@tcp [ 7621.833660] Key type .llcrypt unregistered [ 7621.835470] Key type ._llcrypt unregistered [ 7660.736871] Key type ._llcrypt registered [ 7660.738711] Key type .llcrypt registered [ 7660.814225] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7661.600233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7661.656141] alg: No test for adler32 (adler32-zlib) [ 7662.707954] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 7662.984562] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 7664.711143] Key type lgssc registered [ 7666.109188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7674.562944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7674.590139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7676.039939] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7676.064206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7676.354862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7681.614115] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7690.146237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7690.220029] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7690.244319] Lustre: 255096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7690.576750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7695.797357] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7704.404470] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7711.493391] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7721.135746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7721.532249] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7721.679307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7728.421034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7730.474659] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7739.058302] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7795.176099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7795.181581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7795.191993] Lustre: Skipped 1 previous similar message [ 7795.192857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7795.217675] Lustre: Skipped 1 previous similar message [ 7796.422552] Lustre: server umount lustre-OST0000 complete [ 7800.889559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7813.019135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7813.409971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7815.477266] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7815.488117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7815.495135] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7815.515237] Lustre: Skipped 1 previous similar message [ 7820.033395] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7828.309073] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7871.971290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7871.973512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7872.001145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7872.006687] LustreError: Skipped 2 previous similar messages [ 7872.035137] Lustre: Skipped 1 previous similar message [ 7874.716446] Lustre: server umount lustre-OST0000 complete [ 7880.160300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7880.167525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.183953] Lustre: Skipped 1 previous similar message [ 7880.194082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7882.219795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7882.220506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7882.238296] Lustre: Skipped 1 previous similar message [ 7884.574870] Lustre: server umount lustre-MDT0000 complete [ 7888.560154] LustreError: 255087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799105 with bad export cookie 3147887868588653404 [ 7888.567260] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7888.932400] Lustre: server umount lustre-MDT0001 complete [ 7901.288640] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7904.097500] Key type lgssc unregistered [ 7904.362545] LNet: 260524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7904.371344] LNetError: 260524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7904.384702] LNet: Removed LNI 192.168.201.139@tcp [ 7905.159797] Key type .llcrypt unregistered [ 7905.161784] Key type ._llcrypt unregistered [ 7930.280626] Key type ._llcrypt registered [ 7930.283184] Key type .llcrypt registered [ 7930.383201] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7931.206748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7931.382847] alg: No test for adler32 (adler32-zlib) [ 7932.433616] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 7932.647492] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 7934.255179] Key type lgssc registered [ 7935.755046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7943.745311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7943.768313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7945.105439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7950.040728] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7955.432271] LustreError: 261941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7955.458844] LustreError: 261941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7959.471436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7959.860852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7965.311517] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7973.908721] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7980.692049] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7988.415646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7988.897024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7991.010055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7996.045938] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8004.849319] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8030.181925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8030.184827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8030.207151] Lustre: Skipped 1 previous similar message [ 8030.211935] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8035.312958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8035.317406] Lustre: Skipped 2 previous similar messages [ 8036.152892] Lustre: server umount lustre-OST0000 complete [ 8041.959431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8041.972291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8041.993625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.544836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.554437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8050.665224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8050.670519] Lustre: Skipped 2 previous similar messages [ 8055.263137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8055.532538] Lustre: server umount lustre-MDT0000 complete [ 8055.792565] LustreError: 261941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8055.824974] LustreError: 261941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8060.413092] LustreError: 261923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799277 with bad export cookie 13842458127561922256 [ 8060.422385] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8060.424849] LustreError: 261923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8061.056557] Lustre: server umount lustre-MDT0001 complete [ 8072.039831] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 8074.725778] Key type lgssc unregistered [ 8074.956660] LNet: 266260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8074.964569] LNetError: 266260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8074.994293] LNet: Removed LNI 192.168.201.139@tcp [ 8076.024182] Key type .llcrypt unregistered [ 8076.027216] Key type ._llcrypt unregistered [ 8094.148343] Key type ._llcrypt registered [ 8094.150830] Key type .llcrypt registered [ 8094.281787] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8095.165710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8095.323953] alg: No test for adler32 (adler32-zlib) [ 8096.390225] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 8096.808763] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 8098.528110] Key type lgssc registered [ 8099.811435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8109.644913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8109.687049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8111.504271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8116.274986] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8121.840397] LustreError: 267482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8121.868176] LustreError: 267482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8125.843930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8126.289372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8131.159596] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8139.395969] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8146.284968] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8154.387648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8155.114762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8156.161798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8161.723836] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8170.648341] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8186.854747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8186.864660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8186.868718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8186.883528] LustreError: Skipped 1 previous similar message [ 8191.979325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8191.985435] Lustre: Skipped 2 previous similar messages [ 8197.106301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8197.116288] Lustre: Skipped 1 previous similar message [ 8198.113116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8198.319169] Lustre: server umount lustre-OST0000 complete [ 8202.229549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8202.233505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8202.244347] Lustre: Skipped 2 previous similar messages [ 8207.332510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8207.336521] Lustre: Skipped 2 previous similar messages [ 8208.536804] Lustre: server umount lustre-MDT0000 complete [ 8212.464915] LustreError: 268210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8212.484205] LustreError: 268210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8213.066379] LustreError: 267462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799430 with bad export cookie 14180887916987802253 [ 8213.083248] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8213.094760] LustreError: 267462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8213.497766] Lustre: server umount lustre-MDT0001 complete [ 8225.909213] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 8229.135463] Key type lgssc unregistered [ 8229.441134] LNet: 271167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8229.448882] LNetError: 271167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8230.501809] LNet: Removed LNI 192.168.201.139@tcp [ 8231.274254] Key type .llcrypt unregistered [ 8231.276558] Key type ._llcrypt unregistered [ 8254.509133] Key type ._llcrypt registered [ 8254.510682] Key type .llcrypt registered [ 8254.626969] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8256.414576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8256.536351] alg: No test for adler32 (adler32-zlib) [ 8257.652978] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 8257.992621] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 8259.728310] Key type lgssc registered [ 8261.161107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8269.284455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8269.330807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8270.824165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8275.537429] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8280.046655] LustreError: 272564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8280.061614] LustreError: 272564:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8284.762327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8285.072444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8289.694746] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8297.593338] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8304.004511] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8312.586757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8313.074558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8315.136740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8321.946023] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8331.794463] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8356.321224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8356.334579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8356.346756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8360.423047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8360.425815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8360.443361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8361.968704] Lustre: server umount lustre-OST0000 complete [ 8367.072501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8367.078903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8367.090687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8367.095700] Lustre: Skipped 1 previous similar message [ 8370.656970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8370.658584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8370.674566] Lustre: Skipped 1 previous similar message [ 8375.786415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8375.796903] Lustre: Skipped 1 previous similar message [ 8380.384622] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8380.623978] Lustre: server umount lustre-MDT0000 complete [ 8380.897817] LustreError: 273291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8380.918401] LustreError: 273291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8384.641224] LustreError: 272542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799601 with bad export cookie 5091595714715096334 [ 8384.644589] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8384.648608] LustreError: 272542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8385.089802] Lustre: server umount lustre-MDT0001 complete [ 8396.641390] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 8399.295238] Key type lgssc unregistered [ 8399.601223] LNet: 276876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8399.617628] LNetError: 276876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8400.682314] LNet: Removed LNI 192.168.201.139@tcp [ 8401.478505] Key type .llcrypt unregistered [ 8401.480495] Key type ._llcrypt unregistered [ 8419.535124] Key type ._llcrypt registered [ 8419.536902] Key type .llcrypt registered [ 8419.703469] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8420.521726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8420.649356] alg: No test for adler32 (adler32-zlib) [ 8421.707690] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 8421.921820] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 8423.559510] Key type lgssc registered [ 8424.679767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8432.075725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8432.107200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8433.773147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8439.043939] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8443.383171] LustreError: 278113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8443.395860] LustreError: 278113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8448.488146] LustreError: 278115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8449.148668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8449.663917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8453.941105] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8461.839676] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8468.163182] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8474.650443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8475.002061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8478.066934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8481.856837] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8488.989977] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8503.776875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8503.789255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8503.802498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8504.803541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8504.805600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8504.807891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8504.825643] Lustre: Skipped 1 previous similar message [ 8509.143741] Lustre: server umount lustre-OST0000 complete [ 8515.041558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8515.048182] Lustre: Skipped 1 previous similar message [ 8515.049043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8518.913692] Lustre: server umount lustre-MDT0000 complete [ 8520.163261] LustreError: 278116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8520.174874] LustreError: 278116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8522.237349] LustreError: 278078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799739 with bad export cookie 4347438069434903998 [ 8522.239544] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8522.245504] LustreError: 278078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8522.562509] Lustre: server umount lustre-MDT0001 complete [ 8533.664615] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 8536.213414] Key type lgssc unregistered [ 8536.586642] LNet: 281827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8536.590247] LNetError: 281827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8536.602105] LNet: Removed LNI 192.168.201.139@tcp [ 8537.279766] Key type .llcrypt unregistered [ 8537.282533] Key type ._llcrypt unregistered [ 8550.010455] Key type ._llcrypt registered [ 8550.012471] Key type .llcrypt registered [ 8550.098084] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8568.982682] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 8590.191491] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8591.459982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8591.629048] alg: No test for adler32 (adler32-zlib) [ 8592.817345] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 8593.106070] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 8594.791132] Key type lgssc registered [ 8596.162756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8635.912850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8635.930919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8637.228265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8637.286769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8637.379809] Lustre: lustre-MDT0000: new disk, initializing [ 8637.470899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8637.489901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8642.696366] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8655.178552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8655.346553] Lustre: 285384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8655.410458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8655.418485] Lustre: Skipped 1 previous similar message [ 8655.477312] Lustre: lustre-MDT0001: new disk, initializing [ 8655.601283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8655.640236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8655.658795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8661.198882] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8671.389424] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8677.603325] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8684.830513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8685.273564] Lustre: lustre-OST0000: new disk, initializing [ 8685.278450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8685.291626] Lustre: 287351:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8685.414609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8691.623268] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8693.394566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8693.408741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8693.505359] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8702.734312] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8708.594823] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8708.787334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8714.478568] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8714.764318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8719.330174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8719.346610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8719.381272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8719.386405] Lustre: Skipped 1 previous similar message [ 8724.454514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8724.466930] Lustre: Skipped 1 previous similar message [ 8729.579711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8729.594776] Lustre: Skipped 1 previous similar message [ 8731.103165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8731.289902] Lustre: server umount lustre-OST0000 complete [ 8737.763062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8737.771862] LustreError: Skipped 1 previous similar message [ 8737.775284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8737.779399] Lustre: Skipped 1 previous similar message [ 8737.782161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8739.813133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8742.203386] Lustre: server umount lustre-MDT0000 complete [ 8746.423442] LustreError: 287350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799963 with bad export cookie 18185977964722123864 [ 8746.427220] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8746.436139] LustreError: 287350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8746.784543] Lustre: server umount lustre-MDT0001 complete [ 8756.682586] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:06:13 (1787799973) [ 8764.982651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8765.371079] LustreError: 289766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8765.398594] LustreError: 289766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8765.468097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8770.533580] LustreError: 289767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8770.554564] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8775.655560] LustreError: 289766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8779.374306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8779.862563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8784.961905] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8793.470311] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8799.582502] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8806.908070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8807.505412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8816.181703] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8825.854459] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8835.552797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8835.562201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8835.566510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8835.569458] Lustre: Skipped 1 previous similar message [ 8835.582821] Lustre: Skipped 2 previous similar messages [ 8836.718752] Lustre: server umount lustre-OST0000 complete [ 8840.677073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8842.635662] Lustre: server umount lustre-MDT0000 complete [ 8845.798118] LustreError: 289767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.366249] LustreError: 289746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800063 with bad export cookie 18185977964722124753 [ 8846.383927] LustreError: 289746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8846.385701] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8846.897709] Lustre: server umount lustre-MDT0001 complete [ 8860.068450] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 8864.242775] Key type lgssc unregistered [ 8864.703080] LNet: 293305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8864.710429] LNetError: 293305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8865.766296] LNet: Removed LNI 192.168.201.139@tcp [ 8867.210333] Key type .llcrypt unregistered [ 8867.216099] Key type ._llcrypt unregistered [ 8894.611870] Key type ._llcrypt registered [ 8894.614168] Key type .llcrypt registered [ 8894.695848] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 8907.109222] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8907.642318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8907.732043] alg: No test for adler32 (adler32-zlib) [ 8908.706111] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 8908.922732] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 8910.575190] Key type lgssc registered [ 8911.410721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8960.796828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8960.818648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8962.059096] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8962.087623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8962.155732] Lustre: lustre-MDT0000: new disk, initializing [ 8962.267029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8962.294210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8967.651943] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8977.755922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8977.846679] Lustre: 297113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8977.900847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8977.903720] Lustre: Skipped 1 previous similar message [ 8977.971469] Lustre: lustre-MDT0001: new disk, initializing [ 8978.058818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8978.085360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8978.092662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8982.313348] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8993.141818] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9000.487203] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9009.207874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9009.654765] Lustre: lustre-OST2710: new disk, initializing [ 9009.712104] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9009.722805] Lustre: 299081:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9009.871314] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9017.234601] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9018.440208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9018.452944] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9018.603644] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9026.618891] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9033.286678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9033.506589] Lustre: lustre-OST03e8: new disk, initializing [ 9033.509631] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9033.521905] Lustre: 300146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9033.682760] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9040.733599] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9042.012298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9042.026085] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9042.137828] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9049.186238] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9056.955162] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9057.290237] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9061.823613] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9062.020159] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9066.997427] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9067.254810] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9073.151753] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9073.460980] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9078.243645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9078.250029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9078.254342] Lustre: Skipped 1 previous similar message [ 9078.268129] Lustre: Skipped 3 previous similar messages [ 9080.473131] Lustre: server umount lustre-MDT0000 complete [ 9084.368302] LustreError: 297105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800301 with bad export cookie 17121252290761024848 [ 9084.377710] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9084.377848] LustreError: 297105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9084.851697] Lustre: server umount lustre-MDT0001 complete [ 9098.210665] Lustre: server umount lustre-OST2710 complete [ 9112.608573] Lustre: server umount lustre-OST03e8 complete [ 9131.306283] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 9142.805320] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 9176.366676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9176.583980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9176.618542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9176.726588] Lustre: lustre-MDT0000: new disk, initializing [ 9176.811088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9176.819553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9181.488815] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9191.943983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9192.097430] Lustre: 305399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9192.127230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9192.131149] Lustre: Skipped 1 previous similar message [ 9192.189813] Lustre: lustre-MDT0001: new disk, initializing [ 9192.264338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9192.294399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9192.306809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9196.651277] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9208.354877] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9214.507272] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9222.594675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9223.044573] Lustre: lustre-OST0000: new disk, initializing [ 9223.050361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9223.063540] Lustre: 307368:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9223.220548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9230.463799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9230.480808] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9230.614500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9230.772771] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9241.557924] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9247.302584] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9247.531735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9253.143905] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9253.331734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9256.429646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9256.429710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9256.446341] Lustre: Skipped 3 previous similar messages [ 9256.450802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9256.470839] LustreError: Skipped 1 previous similar message [ 9256.502207] Lustre: Skipped 1 previous similar message [ 9261.545680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9262.036059] Lustre: server umount lustre-OST0000 complete [ 9266.659150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9266.670326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9266.679869] Lustre: Skipped 2 previous similar messages [ 9271.777935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9271.793078] Lustre: Skipped 1 previous similar message [ 9272.100915] Lustre: server umount lustre-MDT0000 complete [ 9276.901075] LustreError: 306266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9276.938541] LustreError: 306266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9276.965924] LustreError: 307367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800493 with bad export cookie 17121252290761026528 [ 9276.970079] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9276.974961] LustreError: 307367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9277.312572] Lustre: server umount lustre-MDT0001 complete [ 9287.031628] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:15:03 (1787800503) [ 9289.325655] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9292.039812] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:15:07 (1787800507) [ 9296.914241] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 9326.680572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9326.989402] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9327.008680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9327.135310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9331.547438] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9341.899082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9342.027947] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9342.077573] Lustre: 311228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9347.669632] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9370.031509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9370.389271] Lustre: Denying initial registration attempt from nid 192.168.201.139@tcp, specified as failover [ 9370.403734] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9370.412389] LustreError: 312681:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9370.423263] LustreError: 312681:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9370.430959] LustreError: 312681:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9370.517518] Lustre: server umount lustre-OST0000 complete [ 9370.521295] LustreError: 312681:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9382.380650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9382.386430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9382.399017] Lustre: Skipped 2 previous similar messages [ 9382.411064] Lustre: Skipped 1 previous similar message [ 9384.869042] Lustre: server umount lustre-MDT0000 complete [ 9387.489846] LustreError: 311238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9387.516525] LustreError: 311238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9389.065718] LustreError: 311220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800606 with bad export cookie 17121252290761027410 [ 9389.072860] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9389.076191] LustreError: 311220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9389.335199] Lustre: server umount lustre-MDT0001 complete [ 9401.765500] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 9404.785890] Key type lgssc unregistered [ 9405.229892] LNet: 313805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9405.236915] LNetError: 313805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9405.253284] LNet: Removed LNI 192.168.201.139@tcp [ 9406.413569] Key type .llcrypt unregistered [ 9406.415981] Key type ._llcrypt unregistered [ 9418.179030] Key type ._llcrypt registered [ 9418.180274] Key type .llcrypt registered [ 9418.298238] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 9419.462164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9419.559365] alg: No test for adler32 (adler32-zlib) [ 9420.788715] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 9421.190381] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 9422.959297] Key type lgssc registered [ 9424.142699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9473.376928] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 9485.349028] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9485.372060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9486.594856] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9486.611528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9486.703705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9492.827545] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9502.738928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9502.868141] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9502.928240] Lustre: 316570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9503.253356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9508.569285] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9517.385812] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9523.701525] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9544.113428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9544.415431] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9544.707756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9552.561300] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9562.274617] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9564.648730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9564.657715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9564.670132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9569.249176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9569.258106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9569.289844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9570.962074] Lustre: server umount lustre-OST0000 complete [ 9580.004285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9580.009574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9580.009920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9580.009926] Lustre: Skipped 1 previous similar message [ 9580.026253] Lustre: Skipped 1 previous similar message [ 9581.790427] Lustre: server umount lustre-MDT0000 complete [ 9585.800369] LustreError: 316563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800802 with bad export cookie 11166321323627507865 [ 9585.806628] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9585.810396] LustreError: 316563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9586.403879] Lustre: server umount lustre-MDT0001 complete [ 9597.488358] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 9600.362375] Key type lgssc unregistered [ 9600.589831] LNet: 320147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9600.594272] LNetError: 320147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9600.605362] LNet: Removed LNI 192.168.201.139@tcp [ 9601.563185] Key type .llcrypt unregistered [ 9601.565939] Key type ._llcrypt unregistered [ 9628.866301] Key type ._llcrypt registered [ 9628.868848] Key type .llcrypt registered [ 9628.959242] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 9630.203850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9630.326671] alg: No test for adler32 (adler32-zlib) [ 9631.300775] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 9631.589346] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 9633.361066] Key type lgssc registered [ 9634.453297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9642.156381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9642.199081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9643.616975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9648.941857] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9653.281599] LustreError: 321566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9653.299100] LustreError: 321566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9658.339075] LustreError: 321565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9660.434739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9660.917752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9665.724564] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9674.389912] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9681.034738] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9691.055032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9691.526402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9697.836823] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9706.832721] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9727.458798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9727.474949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9727.498492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9729.140679] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) [ 9729.514091] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9729.557959] Lustre: Skipped 1 previous similar message [ 9732.973941] Lustre: server umount lustre-MDT0000 complete [ 9734.245910] LustreError: 322855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9736.704647] LustreError: 321547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800953 with bad export cookie 14369137032336785015 [ 9736.717989] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9736.722612] LustreError: 321547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9739.364545] LustreError: 321560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9739.398933] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.39@tcp (stopping) [ 9739.400167] LustreError: 321560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9739.424974] Lustre: Skipped 3 previous similar messages [ 9739.745355] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9741.798141] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9741.809386] Lustre: Skipped 1 previous similar message [ 9743.134386] Lustre: server umount lustre-MDT0001 complete [ 9749.784471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9758.570240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9761.253041] LustreError: 320873:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cd365171 x1874645062041856/t0(0) o250->MGC192.168.201.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9761.766556] LustreError: 323398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9761.787511] LustreError: 323398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9761.889958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9761.957332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9767.513693] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9772.006324] LustreError: 325181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9772.019916] LustreError: 325181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9772.028625] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9777.389632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9777.871106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9783.047912] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9783.291923] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9792.409274] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9794.346621] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9800.910727] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9802.774191] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9812.965582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9812.979462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9813.003633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9824.223239] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9824.238091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9824.256945] Lustre: Skipped 5 previous similar messages [ 9824.497662] Lustre: server umount lustre-OST0000 complete [ 9828.837302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9828.847151] LustreError: Skipped 1 previous similar message [ 9828.851957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9828.869393] Lustre: Skipped 1 previous similar message [ 9835.008544] Lustre: server umount lustre-MDT0000 complete [ 9838.925841] LustreError: 322300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801055 with bad export cookie 14369137032336792855 [ 9838.931491] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9838.943776] LustreError: 322300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9839.434629] Lustre: server umount lustre-MDT0001 complete [ 9853.488691] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 9856.933132] Key type lgssc unregistered [ 9857.353586] LNet: 327864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9857.365667] LNetError: 327864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9857.396116] LNet: Removed LNI 192.168.201.139@tcp [ 9858.518305] Key type .llcrypt unregistered [ 9858.520438] Key type ._llcrypt unregistered [ 9885.282729] Key type ._llcrypt registered [ 9885.289467] Key type .llcrypt registered [ 9885.425102] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 9886.603382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9886.624988] alg: No test for adler32 (adler32-zlib) [ 9888.057401] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [ 9888.312672] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 9889.999227] Key type lgssc registered [ 9891.393029] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9899.912627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9899.921775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9901.383610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9906.743579] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9911.788381] LustreError: 329284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9911.795338] LustreError: 329284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9914.616252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9914.956626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9921.065450] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9931.777484] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9940.139097] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9944.593164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9944.603241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9944.618746] Lustre: Skipped 1 previous similar message [ 9944.629921] Lustre: Skipped 1 previous similar message [ 9949.666805] LustreError: 330012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9949.701122] LustreError: 330012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9949.762308] Lustre: server umount lustre-MDT0000 complete [ 9954.087504] LustreError: 329265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801171 with bad export cookie 15423993156748789685 [ 9954.093466] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9954.099250] LustreError: 329265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9954.474937] Lustre: server umount lustre-MDT0001 complete [ 9968.126442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9968.312543] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9968.395197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9968.739743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9974.234678] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9983.193646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9983.342728] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9983.397783] Lustre: 331875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9983.681585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9988.722281] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9997.973667] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10005.339386] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10015.015975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10015.133676] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10015.143574] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10015.152821] LustreError: 333757:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10015.157531] LustreError: 333757:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10015.161200] LustreError: 333757:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10015.212859] Lustre: server umount lustre-OST0000 complete [10015.215431] LustreError: 333757:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10025.962467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10026.172184] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10026.447970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10033.025318] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10034.382481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10041.931495] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10049.855385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10049.996175] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10050.175217] Lustre: lustre-OST0001: new disk, initializing [10050.180342] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10050.189486] Lustre: 335300:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10050.335706] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10056.143953] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10059.867311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10059.879464] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10059.979624] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10063.002854] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10070.496316] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10070.498336] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10070.509017] LustreError: Skipped 1 previous similar message [10070.521087] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10070.534955] Lustre: Skipped 1 previous similar message [10070.558581] Lustre: Skipped 1 previous similar message [10071.550869] Lustre: server umount lustre-OST0001 complete [10075.630959] LustreError: 334225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10075.641255] LustreError: 334225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10080.225133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10080.227619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10080.238312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10080.742164] LustreError: 334224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10080.773436] LustreError: 334224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10081.980856] Lustre: server umount lustre-OST0000 complete [10085.856747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10085.875432] Lustre: Skipped 1 previous similar message [10085.888917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10085.899638] Lustre: Skipped 2 previous similar messages [10090.980344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10090.993479] Lustre: Skipped 1 previous similar message [10100.191201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10100.192841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10100.212791] Lustre: Skipped 3 previous similar messages [10100.583687] Lustre: server umount lustre-MDT0000 complete [10104.753928] LustreError: 331866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801321 with bad export cookie 15423993156748790238 [10104.765829] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10105.218976] Lustre: server umount lustre-MDT0001 complete [10119.818672] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [10123.405572] Key type lgssc unregistered [10123.630481] LNet: 337263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10123.634927] LNetError: 337263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10123.651632] LNet: Removed LNI 192.168.201.139@tcp [10124.458425] Key type .llcrypt unregistered [10124.460757] Key type ._llcrypt unregistered [10197.145530] Key type ._llcrypt registered [10197.147715] Key type .llcrypt registered [10197.230628] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [10212.264779] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [10213.653117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10213.682880] alg: No test for adler32 (adler32-zlib) [10214.905359] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [10215.207425] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [10216.863883] Key type lgssc registered [10218.374605] Lustre: Echo OBD driver; http://www.lustre.org/ [10256.155629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10256.189542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10257.435661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10257.465839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10257.584646] Lustre: lustre-MDT0000: new disk, initializing [10257.693612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10257.728192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10262.357602] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10273.689836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10273.793932] Lustre: 342200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10273.890601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10273.904609] Lustre: Skipped 1 previous similar message [10274.034094] Lustre: lustre-MDT0001: new disk, initializing [10274.242302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10274.380907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10274.398325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10280.486559] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10291.331186] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10297.343726] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10305.113590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10305.507737] Lustre: lustre-OST0000: new disk, initializing [10305.518983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10305.526727] Lustre: 344167:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10305.641202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10310.750298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10310.755843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10310.900291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10312.365551] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10322.914511] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10328.892172] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10329.265126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10335.069118] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10335.422163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10341.867480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10341.878110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10341.885666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10341.897483] Lustre: Skipped 1 previous similar message [10341.913152] Lustre: Skipped 1 previous similar message [10344.331825] Lustre: server umount lustre-OST0000 complete [10351.079825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10351.089902] LustreError: Skipped 1 previous similar message [10351.098143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10351.113333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10352.102219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10352.142881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10352.148450] Lustre: Skipped 1 previous similar message [10354.822842] Lustre: server umount lustre-MDT0000 complete [10359.356938] LustreError: 342192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801576 with bad export cookie 2095555690033997854 [10359.360728] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10359.369028] LustreError: 342192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10359.952734] Lustre: server umount lustre-MDT0001 complete [10368.628332] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 23:33:05 (1787801585) [10375.691754] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 23:33:12 (1787801592) [10384.209952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10384.709793] LustreError: 346923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10384.739041] LustreError: 346923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10384.803440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10389.995696] LustreError: 346924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10390.135591] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10394.084496] LustreError: 346923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10398.837476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10399.170540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10404.209689] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10412.171995] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10417.798859] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10424.907683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10425.246050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10431.464802] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10439.069185] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10451.511745] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10453.282808] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10454.739434] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10456.199493] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10457.974484] Lustre: Failing over lustre-MDT0000 [10458.209586] Lustre: server umount lustre-MDT0000 complete [10460.129160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10460.139431] LustreError: 346919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10460.142398] Lustre: Skipped 2 previous similar messages [10460.160172] LustreError: 346919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10465.251672] LustreError: 346918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10465.285408] LustreError: 346918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10475.501805] LustreError: 346918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10475.523112] LustreError: 346918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10476.517606] Lustre: 340423:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787801677/real 1787801677] req@00000000c51b2b9c x1874645674433408/t0(0) o400->MGC192.168.201.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787801693 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10476.532790] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10476.701168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10487.102663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10487.162676] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10488.935608] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10491.700207] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10492.398815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10492.445708] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10492.513710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10500.610214] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10502.024709] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10505.171179] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10507.407258] Lustre: Failing over lustre-MDT0000 [10507.721845] Lustre: server umount lustre-MDT0000 complete [10507.744645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10507.750385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10507.768079] Lustre: Skipped 1 previous similar message [10507.775208] LustreError: 346923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10507.803876] LustreError: 346923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [10527.901590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10527.990991] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10528.183131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10528.233898] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10530.914493] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10532.034591] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10533.361566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10533.367436] Lustre: Skipped 2 previous similar messages [10533.382871] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10533.461804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10541.429374] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10543.164942] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10548.716461] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10548.717713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10548.736386] Lustre: Skipped 3 previous similar messages [10548.741393] Lustre: Skipped 1 previous similar message [10553.825980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10553.830548] Lustre: Skipped 2 previous similar messages [10554.523994] Lustre: server umount lustre-MDT0000 complete [10558.247038] LustreError: 347658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801775 with bad export cookie 2095555690034000563 [10558.252619] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10558.263760] LustreError: 347658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10558.579331] Lustre: server umount lustre-MDT0001 complete [10573.158488] Lustre: server umount lustre-OST0000 complete [10589.044483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10589.508961] LustreError: 353283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10589.528410] LustreError: 353283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [10593.042909] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10601.707887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10602.040513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10602.054440] Lustre: Skipped 1 previous similar message [10606.847316] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10614.337356] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10619.658456] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10627.102387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10633.714515] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10634.639630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10642.651649] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10655.205598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10655.218345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10655.235238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10659.417930] Lustre: server umount lustre-OST0000 complete [10663.395286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10663.403910] LustreError: Skipped 1 previous similar message [10669.818472] Lustre: server umount lustre-MDT0000 complete [10672.617433] LustreError: 353283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10672.648882] LustreError: 353283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10673.858129] LustreError: 353264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801890 with bad export cookie 2095555690034001312 [10673.872308] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10674.132555] Lustre: server umount lustre-MDT0001 complete [10686.504655] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [10689.694579] Key type lgssc unregistered [10690.096096] LNet: 356818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10690.112978] LNetError: 356818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10691.175666] LNet: Removed LNI 192.168.201.139@tcp [10692.021814] Key type .llcrypt unregistered [10692.023545] Key type ._llcrypt unregistered [10719.035949] Key type ._llcrypt registered [10719.038953] Key type .llcrypt registered [10719.148948] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [10720.570537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10720.598384] alg: No test for adler32 (adler32-zlib) [10721.767487] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [10722.086471] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [10723.783224] Key type lgssc registered [10725.268439] Lustre: Echo OBD driver; http://www.lustre.org/ [10734.088693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10734.121461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10735.501266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10740.063415] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10745.825662] LustreError: 358335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10745.845323] LustreError: 358335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10749.253669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10749.923821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10755.209326] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10763.459245] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10769.886641] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10778.184861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10779.142828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10780.163656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10790.824229] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10802.675474] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10814.953493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10814.962840] LustreError: 358332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10814.966904] Lustre: Skipped 2 previous similar messages [10814.984771] LustreError: 358332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10815.129534] Lustre: server umount lustre-MDT0000 complete [10819.489743] LustreError: 358318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802036 with bad export cookie 10411612115495993870 [10819.501139] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10819.981158] Lustre: server umount lustre-MDT0001 complete [10834.985463] Lustre: server umount lustre-OST0000 complete [10848.673730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10849.224311] LustreError: 362049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10849.240321] LustreError: 362049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10849.292884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10854.377050] LustreError: 362049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10854.582154] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10859.493369] LustreError: 362050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10864.349475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10864.608976] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10864.861208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10871.197900] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10879.423751] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10886.444026] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10898.280281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10898.521534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10905.465691] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10913.127629] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10919.041119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10939.879427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10939.891093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10939.896390] Lustre: Skipped 2 previous similar messages [10939.912412] Lustre: Skipped 2 previous similar messages [10944.773866] Lustre: server umount lustre-MDT0000 complete [10944.994470] LustreError: 364543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10945.016930] LustreError: 364543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10949.534176] LustreError: 362031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802166 with bad export cookie 10411612115495994983 [10949.547719] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10949.551877] LustreError: 362031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10949.852513] Lustre: server umount lustre-MDT0001 complete [10965.120987] Lustre: server umount lustre-OST0000 complete [10983.563765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10984.027091] LustreError: 365987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10984.107775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10989.189260] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10999.675876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11006.351825] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11017.667438] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11024.608724] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11032.544392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11032.973873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11032.982743] Lustre: Skipped 1 previous similar message [11040.736266] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11049.820748] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11054.530766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11064.802460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11064.808167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11064.810181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11064.841382] Lustre: Skipped 1 previous similar message [11067.290764] Lustre: server umount lustre-OST0000 complete [11075.050093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11075.059790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11075.066748] Lustre: Skipped 1 previous similar message [11075.082618] Lustre: Skipped 2 previous similar messages [11078.399370] Lustre: server umount lustre-MDT0000 complete [11080.170067] LustreError: 365983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11080.201778] LustreError: 365983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11083.010871] LustreError: 365967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802299 with bad export cookie 10411612115495996005 [11083.015277] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11083.021703] LustreError: 365967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11083.521529] Lustre: server umount lustre-MDT0001 complete [11095.178869] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [11098.229977] Key type lgssc unregistered [11098.554990] LNet: 369524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11098.558735] LNetError: 369524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11099.626940] LNet: Removed LNI 192.168.201.139@tcp [11101.119330] Key type .llcrypt unregistered [11101.121328] Key type ._llcrypt unregistered [11130.021782] Key type ._llcrypt registered [11130.027748] Key type .llcrypt registered [11130.234605] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [11131.567115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11131.799466] alg: No test for adler32 (adler32-zlib) [11133.411620] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [11133.863430] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [11135.823431] Key type lgssc registered [11138.104847] Lustre: Echo OBD driver; http://www.lustre.org/ [11149.464304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11149.502932] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11149.517370] LustreError: 370966:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11149.603062] LustreError: 370966:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11149.620189] LustreError: 370966:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11149.629765] LustreError: 370966:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11149.671066] LustreError: 370966:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11158.225096] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11158.232876] LustreError: 371363:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11158.248133] LustreError: 371363:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11158.255989] LustreError: 371363:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11158.263119] LustreError: 371363:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11158.274226] LustreError: 371363:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11183.503882] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [11186.955788] Key type lgssc unregistered [11187.271532] LNet: 372370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11187.275970] LNetError: 372370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11187.317881] LNet: Removed LNI 192.168.201.139@tcp [11188.491323] Key type .llcrypt unregistered [11188.492726] Key type ._llcrypt unregistered [11213.536300] Key type ._llcrypt registered [11213.543826] Key type .llcrypt registered [11213.746389] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [11232.231563] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [11233.928651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11234.026739] alg: No test for adler32 (adler32-zlib) [11235.107454] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [11235.373850] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [11237.079197] Key type lgssc registered [11238.821365] Lustre: Echo OBD driver; http://www.lustre.org/ [11280.538955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11280.562974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11281.798507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11281.822982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11281.910994] Lustre: lustre-MDT0000: new disk, initializing [11281.998434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11282.014496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11287.070201] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11299.294343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11299.484201] Lustre: 375529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11299.580676] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11299.585637] Lustre: Skipped 1 previous similar message [11299.646719] Lustre: lustre-MDT0001: new disk, initializing [11299.711867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11299.739459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11299.748785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11305.507829] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11318.374254] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11324.894811] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11333.668123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11334.071315] Lustre: lustre-OST0000: new disk, initializing [11334.076416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11334.091173] Lustre: 377500:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11334.263885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11340.390128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11340.403668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11340.527079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11341.308752] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11353.032645] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11359.846564] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11360.233495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11366.261514] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11366.568518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11371.498797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11371.500908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11371.514144] LustreError: Skipped 1 previous similar message [11371.516750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11371.539801] Lustre: Skipped 1 previous similar message [11374.827226] Lustre: server umount lustre-OST0000 complete [11381.729052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11381.745471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11381.754791] Lustre: Skipped 2 previous similar messages [11385.918392] Lustre: server umount lustre-MDT0000 complete [11390.283657] LustreError: 375522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802607 with bad export cookie 2319917711020195189 [11390.297752] LustreError: 375522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11390.300434] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11390.690724] Lustre: server umount lustre-MDT0001 complete [11399.933781] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11401.624948] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 23:50:18 (1787802618) [11411.309974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11411.790954] LustreError: 379964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11411.821036] LustreError: 379964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11411.979279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11416.453966] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11417.066642] LustreError: 379965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11421.164754] LustreError: 379964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11426.297732] LustreError: 379965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11428.583586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11429.007318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11434.302331] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11444.078154] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11450.565194] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11459.438395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11460.001841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11468.390852] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11479.528191] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11488.588501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11488.804252] Lustre: lustre-OST0001: new disk, initializing [11488.808334] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11488.814192] Lustre: 382804:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11488.907213] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11495.869856] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11496.089843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11496.094140] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11496.242307] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11508.323977] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11515.558368] Lustre: lustre-OST0001: Not available for connect from 192.168.201.39@tcp (stopping) [11516.897085] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11516.898607] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11516.906800] LustreError: Skipped 1 previous similar message [11516.944341] Lustre: Skipped 2 previous similar messages [11519.867659] Lustre: server umount lustre-OST0001 complete [11520.616225] LustreError: 382320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11532.260624] LustreError: 381786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11532.283821] LustreError: 381786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11539.936732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11539.943130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11539.955470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11539.959834] Lustre: Skipped 2 previous similar messages [11545.711192] Lustre: server umount lustre-OST0000 complete [11551.714912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11551.720375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11551.739618] Lustre: Skipped 1 previous similar message [11551.750824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11551.756143] Lustre: Skipped 2 previous similar messages [11556.302288] Lustre: server umount lustre-MDT0000 complete [11557.863795] LustreError: 379964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11557.890076] LustreError: 379964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11560.009634] LustreError: 379944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802777 with bad export cookie 2319917711020196071 [11560.016625] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11560.021253] LustreError: 379944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11566.745765] Lustre: server umount lustre-MDT0001 complete [11579.345254] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [11582.748892] Key type lgssc unregistered [11583.321556] LNet: 384796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11583.352909] LNetError: 384796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11583.397459] LNet: Removed LNI 192.168.201.139@tcp [11584.606696] Key type .llcrypt unregistered [11584.610354] Key type ._llcrypt unregistered [11644.824777] Key type ._llcrypt registered [11644.834640] Key type .llcrypt registered [11645.001410] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [11646.495256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11646.528354] alg: No test for adler32 (adler32-zlib) [11647.470251] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [11647.634158] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [11649.257187] Key type lgssc registered [11650.822798] Lustre: Echo OBD driver; http://www.lustre.org/ [11660.401919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11660.417254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11661.628421] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11661.649463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11661.784830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11667.401439] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11676.377592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11676.554338] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11676.605909] Lustre: 387594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11676.943468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11682.187163] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11689.677199] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11695.244533] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11700.199105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11700.208798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11700.227971] Lustre: Skipped 1 previous similar message [11700.250443] Lustre: Skipped 1 previous similar message [11704.311130] Lustre: server umount lustre-MDT0000 complete [11708.395938] LustreError: 388346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802925 with bad export cookie 17582595445915085580 [11708.413940] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11708.753350] Lustre: server umount lustre-MDT0001 complete [11720.180277] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 23:55:36 (1787802936) [11727.718841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11728.083542] LustreError: 390108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11728.104717] LustreError: 390108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11728.163881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11733.478597] LustreError: 390109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11733.590824] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11738.593529] LustreError: 390108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11742.725758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11743.051070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11748.338379] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11756.072000] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11762.826693] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11769.964446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11770.108900] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11770.275326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11776.486624] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11783.429368] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11791.660293] Lustre: Permanently deactivating lustre-OST0000 [11791.688068] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11797.587175] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11799.782489] LustreError: 393046:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11799.789761] LustreError: 393046:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11799.795768] LustreError: 393046:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11804.183475] LustreError: 393095:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11804.193582] LustreError: 393095:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11812.268360] Lustre: server umount lustre-OST0000 complete [11815.340481] LustreError: 393346:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11815.346249] LustreError: 393346:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11818.983365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11819.001753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11819.012364] Lustre: Skipped 1 previous similar message [11824.097521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11827.229590] Lustre: server umount lustre-MDT0000 complete [11829.218598] LustreError: 390108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11829.232172] LustreError: 390108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11832.066150] LustreError: 390089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803049 with bad export cookie 17582595445915086161 [11832.067803] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11832.079108] LustreError: 390089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11842.981357] Lustre: server umount lustre-MDT0001 complete [11854.559307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11859.177779] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11862.042517] LustreError: 394506:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11868.671466] Lustre: 394846:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.139@tcp [11871.662517] Lustre: server umount MGS complete [11884.885419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11885.473189] LustreError: 395571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11885.498863] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11885.508736] Lustre: Skipped 1 previous similar message [11885.601980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11889.144609] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11895.781225] LustreError: 395572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11895.816417] LustreError: 395572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11900.024557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11900.212247] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11905.757678] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11914.563665] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11919.655593] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11921.988086] Lustre: Permanently reactivating lustre-OST0000 [11922.001782] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11922.008168] Lustre: Skipped 2 previous similar messages [11952.607206] LustreError: 397411:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11961.296903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11961.755780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11961.767316] Lustre: Skipped 1 previous similar message [11969.015696] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11976.701495] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11983.327858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11983.335988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11983.338714] LustreError: Skipped 1 previous similar message [11983.341950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11983.349218] Lustre: Skipped 2 previous similar messages [11983.370112] Lustre: Skipped 2 previous similar messages [11987.505887] Lustre: server umount lustre-OST0000 complete [11992.548596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11992.555402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11992.588092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12003.822825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12003.833932] Lustre: Skipped 5 previous similar messages [12005.343125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12005.537845] Lustre: server umount lustre-MDT0000 complete [12008.938763] LustreError: 396299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12008.966207] LustreError: 396299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12009.591576] LustreError: 395551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803226 with bad export cookie 17582595445915087211 [12009.592992] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12009.616792] LustreError: 395551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12010.130563] Lustre: server umount lustre-MDT0001 complete [12024.390066] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12028.773343] Key type lgssc unregistered [12029.146928] LNet: 399993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12029.151889] LNetError: 399993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12029.165225] LNet: Removed LNI 192.168.201.139@tcp [12030.189257] Key type .llcrypt unregistered [12030.190730] Key type ._llcrypt unregistered [12050.208718] Key type ._llcrypt registered [12050.211380] Key type .llcrypt registered [12050.352654] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [12066.545968] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12067.920359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12068.050485] alg: No test for adler32 (adler32-zlib) [12069.222519] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [12069.539073] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [12071.303732] Key type lgssc registered [12073.279767] Lustre: Echo OBD driver; http://www.lustre.org/ [12115.553688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12115.571828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12116.860353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12116.878370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12116.956487] Lustre: lustre-MDT0000: new disk, initializing [12117.099537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12117.119265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12121.439310] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12132.836318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12132.921132] Lustre: 403167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12132.964363] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12132.970507] Lustre: Skipped 1 previous similar message [12133.024920] Lustre: lustre-MDT0001: new disk, initializing [12133.108430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12133.176127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12133.188111] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12137.981816] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12147.115638] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12152.089919] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12158.159325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12158.451834] Lustre: lustre-OST0000: new disk, initializing [12158.455198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12158.462417] Lustre: 405137:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12158.544950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12165.215962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12165.223788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12165.393543] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12165.411728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12177.163600] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12183.708345] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12183.896887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12188.978856] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12189.210153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12196.324016] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12196.326424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12196.328800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12196.346028] Lustre: Skipped 1 previous similar message [12197.722815] Lustre: server umount lustre-OST0000 complete [12205.030491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12205.073595] LustreError: Skipped 1 previous similar message [12205.084682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12205.143698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12205.157169] Lustre: Skipped 1 previous similar message [12206.560562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12206.562379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12206.595974] Lustre: Skipped 1 previous similar message [12209.369638] Lustre: server umount lustre-MDT0000 complete [12213.510874] LustreError: 403158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803430 with bad export cookie 16104874973079723597 [12213.511281] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12213.523754] LustreError: 403158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12213.875210] Lustre: server umount lustre-MDT0001 complete [12225.888555] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12228.224446] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:04:04 (1787803444) [12239.196577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12240.360786] LustreError: 407598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12240.401515] LustreError: 407598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12240.541512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12245.991035] LustreError: 407599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12246.830521] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12250.092448] LustreError: 407598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12255.204420] LustreError: 407599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12257.769561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12258.422492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12263.662886] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12273.458991] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12279.697835] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12286.600869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12287.105279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12293.927908] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12305.332689] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12309.615461] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12313.689912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12324.325416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12324.329621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12324.335776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12329.239902] Lustre: server umount lustre-OST0000 complete [12335.074224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12335.083037] LustreError: Skipped 1 previous similar message [12335.085289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12335.102922] Lustre: Skipped 1 previous similar message [12335.109803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12335.118425] Lustre: Skipped 2 previous similar messages [12339.909525] Lustre: server umount lustre-MDT0000 complete [12340.715004] LustreError: 407599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12340.742521] LustreError: 407599:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12343.318117] LustreError: 407580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803560 with bad export cookie 16104874973079724479 [12343.325159] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12343.330297] LustreError: 407580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12343.615862] Lustre: server umount lustre-MDT0001 complete [12356.154282] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12361.187722] Key type lgssc unregistered [12361.775164] LNet: 411329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12361.787144] LNetError: 411329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12361.816929] LNet: Removed LNI 192.168.201.139@tcp [12363.164605] Key type .llcrypt unregistered [12363.166564] Key type ._llcrypt unregistered [12397.368735] Key type ._llcrypt registered [12397.370856] Key type .llcrypt registered [12397.533763] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12397.816513] Key type .llcrypt unregistered [12397.818434] Key type ._llcrypt unregistered [12416.692984] Key type ._llcrypt registered [12416.695475] Key type .llcrypt registered [12416.901642] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12417.931223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12418.058488] alg: No test for adler32 (adler32-zlib) [12419.216985] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [12419.471420] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [12421.135158] Key type lgssc registered [12422.247437] Lustre: Echo OBD driver; http://www.lustre.org/ [12430.559732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12430.579405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12431.970953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12437.449853] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12441.060955] LustreError: 413561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12441.093487] LustreError: 413561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12445.924976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12446.178563] LustreError: 413560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12446.283963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12451.300982] LustreError: 413561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12451.315927] LustreError: 413561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12453.591870] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12456.418920] LustreError: 413557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12456.430211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12456.440682] LustreError: 413557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12461.551734] LustreError: 413555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12461.571509] LustreError: 413555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12461.666828] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12468.358924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12468.804642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12474.083970] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12484.065400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12484.073807] Lustre: Skipped 1 previous similar message [12484.073952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12484.086881] Lustre: Skipped 1 previous similar message [12488.979784] Lustre: server umount lustre-OST0000 complete [12494.311461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12494.312507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12494.326967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12494.354566] Lustre: Skipped 1 previous similar message [12499.425188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12499.446653] Lustre: Skipped 2 previous similar messages [12500.060243] Lustre: server umount lustre-MDT0000 complete [12504.566140] LustreError: 413560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12504.595614] LustreError: 413560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12505.635439] LustreError: 413541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803722 with bad export cookie 10688047435310757574 [12505.637866] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12505.644876] LustreError: 413541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12506.234701] Lustre: server umount lustre-MDT0001 complete [12519.023972] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12521.791186] Key type lgssc unregistered [12522.124310] LNet: 416712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12522.127438] LNetError: 416712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12522.139608] LNet: Removed LNI 192.168.201.139@tcp [12523.092110] Key type .llcrypt unregistered [12523.094299] Key type ._llcrypt unregistered [12554.412631] Key type ._llcrypt registered [12554.414654] Key type .llcrypt registered [12554.646385] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12555.768332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12555.804634] alg: No test for adler32 (adler32-zlib) [12556.978886] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [12557.373357] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [12559.136106] Key type lgssc registered [12560.886502] Lustre: Echo OBD driver; http://www.lustre.org/ [12569.268795] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12569.298129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12584.803152] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12589.919232] LustreError: 418104:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.139@tcp: failed processing log, type 4: rc = -110 [12616.735617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12627.375950] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12637.131129] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12647.122429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12647.644425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12647.726675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12654.373426] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12657.121498] LustreError: 419141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12657.136864] LustreError: 419141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12662.242253] LustreError: 418129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12662.262733] LustreError: 418129:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12664.847485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12665.409437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12669.202563] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12677.763203] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12683.702113] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12693.985074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12693.989800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12694.000869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12696.034225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12696.047212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12697.822561] Lustre: server umount lustre-OST0000 complete [12706.277352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12706.277632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12706.288931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12706.291749] Lustre: Skipped 1 previous similar message [12706.306897] Lustre: Skipped 2 previous similar messages [12707.697261] Lustre: server umount lustre-MDT0000 complete [12711.122675] LustreError: 418112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803928 with bad export cookie 5062201713056338300 [12711.132600] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12711.134862] LustreError: 418112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12711.639472] Lustre: server umount lustre-MDT0001 complete [12727.792200] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12732.365130] Key type lgssc unregistered [12732.815872] LNet: 421680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12732.827361] LNetError: 421680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12732.840871] LNet: Removed LNI 192.168.201.139@tcp [12733.749844] Key type .llcrypt unregistered [12733.755246] Key type ._llcrypt unregistered [12759.811798] Key type ._llcrypt registered [12759.813523] Key type .llcrypt registered [12759.952704] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12761.232856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12761.278091] alg: No test for adler32 (adler32-zlib) [12762.467461] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [12762.647449] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [12764.295158] Key type lgssc registered [12765.549383] Lustre: Echo OBD driver; http://www.lustre.org/ [12777.897282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12777.923131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12779.819962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12785.766382] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12789.219110] LustreError: 423100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12789.253205] LustreError: 423100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12794.357408] LustreError: 423099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12797.525511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12798.059712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12805.754503] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12814.438065] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12821.152977] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12828.389548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12828.858422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12834.441775] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12842.460889] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12849.350303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12864.481549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12864.505087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12864.539872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12866.690836] Lustre: server umount lustre-MDT0000 complete [12868.719084] LustreError: 423079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.39@tcp arrived at 1787804085 with bad export cookie 14673492710599358138 [12868.724443] LustreError: 423079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12870.119181] LustreError: 423829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12870.131529] LustreError: 423829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12871.026087] LustreError: 423081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804088 with bad export cookie 14673492710599357473 [12871.048556] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12871.057570] LustreError: 423081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12874.209763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12874.214443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12874.220409] Lustre: Skipped 2 previous similar messages [12874.223403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12874.227683] Lustre: Skipped 3 previous similar messages [12875.232408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12875.232791] LustreError: 423099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12877.271915] Lustre: server umount lustre-OST0000 complete [12881.379942] LustreError: 423829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12881.412428] LustreError: 423829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12884.088671] Lustre: server umount lustre-MDT0001 complete [12894.672552] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12896.905552] Key type lgssc unregistered [12897.135865] LNet: 426812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12897.140384] LNetError: 426812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12897.173142] LNet: Removed LNI 192.168.201.139@tcp [12897.966525] Key type .llcrypt unregistered [12897.968295] Key type ._llcrypt unregistered [12922.837210] Key type ._llcrypt registered [12922.838800] Key type .llcrypt registered [12922.909823] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12924.165892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12924.188129] alg: No test for adler32 (adler32-zlib) [12925.261709] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [12925.547811] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [12927.247763] Key type lgssc registered [12928.904448] Lustre: Echo OBD driver; http://www.lustre.org/ [12940.207300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12940.227859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12941.727532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12947.662477] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12951.018249] LustreError: 428229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12951.024408] LustreError: 428229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12956.133055] LustreError: 428228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12958.358692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12959.171121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12964.349945] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12974.890288] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12982.413955] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12991.270128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12991.644793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12998.418173] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13005.698818] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13011.151653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:193) [13015.521274] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13015.525238] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13015.531246] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13016.546953] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13016.550831] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13017.741878] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.39@tcp (stopping) [13017.756215] Lustre: Skipped 1 previous similar message [13021.664785] LustreError: 428957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13021.676229] LustreError: 428957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13021.818886] Lustre: server umount lustre-MDT0001 complete [13026.793151] LustreError: 428223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13026.843020] LustreError: 428223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13031.904423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13031.905292] LustreError: 428957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13031.957580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13034.583986] Lustre: server umount lustre-OST0000 complete [13039.145674] Lustre: server umount lustre-MDT0000 complete [13060.247429] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [13063.621799] Key type lgssc unregistered [13063.865943] LNet: 431940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13063.875432] LNetError: 431940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13063.893863] LNet: Removed LNI 192.168.201.139@tcp [13064.762253] Key type .llcrypt unregistered [13064.765277] Key type ._llcrypt unregistered [13089.373903] Key type ._llcrypt registered [13089.376230] Key type .llcrypt registered [13089.493882] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [13113.966375] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [13114.912213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13114.931702] alg: No test for adler32 (adler32-zlib) [13115.895274] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [13116.200741] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [13117.920566] Key type lgssc registered [13119.357257] Lustre: Echo OBD driver; http://www.lustre.org/ [13161.109516] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [13177.273385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13177.306378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13178.534547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13178.555821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13178.629359] Lustre: lustre-MDT0000: new disk, initializing [13178.738852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13178.790753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13182.993486] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13200.063810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13200.435227] Lustre: lustre-OST0000: new disk, initializing [13200.436694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13200.438491] Lustre: Skipped 1 previous similar message [13200.440522] Lustre: 436976:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13200.481243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13207.747958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13207.754889] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13207.758870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13207.791959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13220.182464] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13236.896965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13237.034557] Lustre: 435957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13237.126701] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13237.189760] Lustre: lustre-MDT0001: new disk, initializing [13237.361143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13237.460174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13237.483879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13242.125587] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13251.431144] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13251.644411] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13256.536176] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13256.756721] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13262.307643] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13262.325646] Lustre: Skipped 1 previous similar message [13262.326858] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13262.340621] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13262.343636] Lustre: Skipped 1 previous similar message [13267.426140] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13267.429984] Lustre: Skipped 2 previous similar messages [13267.842929] Lustre: server umount lustre-MDT0001 complete [13277.666519] LustreError: 435970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13277.685702] LustreError: 435970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13282.800241] LustreError: 435969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13282.833185] LustreError: 435969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13284.833315] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13284.837988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13284.849423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13284.853160] Lustre: Skipped 1 previous similar message [13287.905913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13287.924962] LustreError: 435970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13293.055480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13293.058079] LustreError: 435969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13298.149127] LustreError: 435970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13299.169108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13299.477631] Lustre: server umount lustre-OST0000 complete [13309.449665] Lustre: server umount lustre-MDT0000 complete [13322.567884] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [13324.874575] Key type lgssc unregistered [13325.064639] LNet: 440726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13325.069734] LNetError: 440726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13325.082600] LNet: Removed LNI 192.168.201.139@tcp [13325.890447] Key type .llcrypt unregistered [13325.892125] Key type ._llcrypt unregistered [13358.326900] Key type ._llcrypt registered [13358.329665] Key type .llcrypt registered [13358.419981] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [13358.721740] Key type .llcrypt unregistered [13358.723471] Key type ._llcrypt unregistered [13372.302609] Key type ._llcrypt registered [13372.304665] Key type .llcrypt registered [13372.398101] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [13373.264907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13373.402248] alg: No test for adler32 (adler32-zlib) [13374.419544] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [13374.661564] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [13376.329268] Key type lgssc registered [13377.313517] Lustre: Echo OBD driver; http://www.lustre.org/ [13400.382518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13400.412323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13401.621421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13401.643778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13401.725596] Lustre: lustre-MDT0000: new disk, initializing [13401.794743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13401.813125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13405.459267] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13414.185035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13414.342859] Lustre: 444084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13414.392575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13414.396572] Lustre: Skipped 1 previous similar message [13414.399930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13414.483517] Lustre: lustre-MDT0001: new disk, initializing [13414.574987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13414.613543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13414.635744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13417.858145] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13425.125123] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13431.219772] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13438.286653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13438.451896] Lustre: lustre-OST0000: new disk, initializing [13438.453980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13438.458646] Lustre: 446052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13438.518536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13444.235782] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13445.735662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13445.748493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13445.835493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13453.235865] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13460.960759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13460.971571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13461.017720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13461.475511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13465.364519] Lustre: server umount lustre-MDT0000 complete [13468.776070] LustreError: 444894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804685 with bad export cookie 11099654594953964434 [13468.778888] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13468.780925] LustreError: 444894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13469.081689] Lustre: server umount lustre-MDT0001 complete [13482.929443] Lustre: server umount lustre-OST0000 complete [13491.865663] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:25:08 (1787804708) [13500.787729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13501.244965] LustreError: 448081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13501.261869] LustreError: 448081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13501.335059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13506.217988] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13506.532223] LustreError: 448082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13511.650081] LustreError: 448081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13513.931940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13514.212254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13519.599665] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13525.967974] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13530.606220] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13550.889995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13551.102623] Lustre: Found index 0 for lustre-OST0000, updating log [13551.290513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13557.526913] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13565.404033] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13572.792941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13583.331771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13583.346039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13583.355863] Lustre: Skipped 2 previous similar messages [13583.372077] Lustre: Skipped 4 previous similar messages [13586.754383] Lustre: server umount lustre-OST0000 complete [13605.865272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13605.880982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13605.883474] Lustre: Skipped 1 previous similar message [13605.899545] Lustre: Skipped 3 previous similar messages [13609.351644] Lustre: server umount lustre-MDT0000 complete [13610.017577] LustreError: 448081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13611.232133] LNet: 1 local NIs in recovery (showing 1): 192.168.201.139@tcp [13612.866349] LustreError: 448063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804829 with bad export cookie 11099654594953965820 [13612.878696] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13613.181817] Lustre: server umount lustre-MDT0001 complete [13621.083983] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:27:18 (1787804838) [14237.264867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14237.599132] LustreError: 454515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14237.614923] LustreError: 454515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [14237.665508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14241.343321] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [14242.806388] LustreError: 454516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14247.761284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14247.906755] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [14248.070880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14251.535805] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [14257.242731] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14261.460733] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14279.020461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14279.171264] Lustre: Found index 0 for lustre-OST0000, updating log [14279.349479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14283.052362] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [14286.508430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [14289.639338] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14291.174505] LNet: 1 local NIs in recovery (showing 1): 192.168.201.139@tcp [14291.946408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14291.960299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14291.970167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14293.435144] Lustre: server umount lustre-OST0000 complete [14297.571459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14297.583833] Lustre: Skipped 1 previous similar message [14297.584153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14299.685082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14299.689018] Lustre: Skipped 4 previous similar messages [14302.753897] LustreError: 454512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14302.786378] Lustre: server umount lustre-MDT0000 complete [14304.801039] LustreError: 454515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14304.816114] LustreError: 454515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14304.894850] LustreError: 454495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805521 with bad export cookie 11099654594953966912 [14304.897136] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14304.909461] LustreError: 454495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [14305.078649] Lustre: server umount lustre-MDT0001 complete [14327.076313] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [14333.259539] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [14360.293907] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 00:39:37 (1787805577) [14377.697672] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [14380.272451] Key type lgssc unregistered [14380.544508] LNet: 461146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14380.550328] LNetError: 461146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14501.426805] LNet: Removed LNI 192.168.201.139@tcp [14502.208685] Key type .llcrypt unregistered [14502.212076] Key type ._llcrypt unregistered [14518.674980] Key type ._llcrypt registered [14518.681072] Key type .llcrypt registered [14518.786100] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [14520.088424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14520.253795] alg: No test for adler32 (adler32-zlib) [14521.343802] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [14521.660282] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [14523.474068] Key type lgssc registered [14524.558425] Lustre: Echo OBD driver; http://www.lustre.org/ [14528.974638] LNet: Added LNI 192.168.201.139@tcp100 [8/256/0/180] [14529.640371] LNet: Added LNI 192.168.201.139@tcp101 [8/256/0/180] [14530.537592] LNet: Added LNI 192.168.201.139@tcp102 [8/256/0/180] [14531.492948] LNet: Added LNI 192.168.201.139@tcp103 [8/256/0/180] [14532.451042] LNet: Added LNI 192.168.201.139@tcp104 [8/256/0/180] [14533.608679] LNet: Added LNI 192.168.201.139@tcp105 [8/256/0/180] [14534.643512] LNet: Added LNI 192.168.201.139@tcp106 [8/256/0/180] [14535.682723] LNet: Added LNI 192.168.201.139@tcp107 [8/256/0/180] [14536.653329] LNet: Added LNI 192.168.201.139@tcp108 [8/256/0/180] [14537.517284] LNet: Added LNI 192.168.201.139@tcp109 [8/256/0/180] [14538.520494] LNet: Added LNI 192.168.201.139@tcp110 [8/256/0/180] [14539.381650] LNet: Added LNI 192.168.201.139@tcp111 [8/256/0/180] [14540.329196] LNet: Added LNI 192.168.201.139@tcp112 [8/256/0/180] [14541.450461] LNet: Added LNI 192.168.201.139@tcp113 [8/256/0/180] [14542.647838] LNet: Added LNI 192.168.201.139@tcp114 [8/256/0/180] [14543.687349] LNet: Added LNI 192.168.201.139@tcp115 [8/256/0/180] [14544.686072] LNet: Added LNI 192.168.201.139@tcp116 [8/256/0/180] [14545.642789] LNet: Added LNI 192.168.201.139@tcp117 [8/256/0/180] [14546.830713] LNet: Added LNI 192.168.201.139@tcp118 [8/256/0/180] [14547.690563] LNet: Added LNI 192.168.201.139@tcp119 [8/256/0/180] [14548.673803] LNet: Added LNI 192.168.201.139@tcp120 [8/256/0/180] [14549.642984] LNet: Added LNI 192.168.201.139@tcp121 [8/256/0/180] [14550.444684] LNet: Added LNI 192.168.201.139@tcp122 [8/256/0/180] [14551.425461] LNet: Added LNI 192.168.201.139@tcp123 [8/256/0/180] [14552.518327] LNet: Added LNI 192.168.201.139@tcp124 [8/256/0/180] [14553.500910] LNet: Added LNI 192.168.201.139@tcp125 [8/256/0/180] [14554.448355] LNet: Added LNI 192.168.201.139@tcp126 [8/256/0/180] [14555.294908] LNet: Added LNI 192.168.201.139@tcp127 [8/256/0/180] [14556.245560] LNet: Added LNI 192.168.201.139@tcp128 [8/256/0/180] [14557.098746] LNet: Added LNI 192.168.201.139@tcp129 [8/256/0/180] [14557.930806] LNet: Added LNI 192.168.201.139@tcp130 [8/256/0/180] [14558.851249] LNet: Added LNI 192.168.201.139@tcp131 [8/256/0/180] [14559.623271] LNet: Added LNI 192.168.201.139@tcp132 [8/256/0/180] [14560.687371] LNet: Added LNI 192.168.201.139@tcp133 [8/256/0/180] [14561.455435] LNet: Added LNI 192.168.201.139@tcp134 [8/256/0/180] [14562.384097] LNet: Added LNI 192.168.201.139@tcp135 [8/256/0/180] [14569.231617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14569.288857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14570.491527] Lustre: 464180:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.139@tcp: can't add NID 192.168.201.139@tcp130: rc = -75 [14570.621348] LNetError: 461639:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.139 [14570.632716] LNetError: Protocol error connecting to 192.168.201.139@tcp118 at host 192.168.201.139:988: is it running a compatible version of Lustre? [14570.669349] LNetError: Skipped 8 previous similar messages [14570.777611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14570.826683] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14570.847651] LNet: 8 local NIs in recovery (showing 8): 192.168.201.139@tcp121, 192.168.201.139@tcp122, 192.168.201.139@tcp124, 192.168.201.139@tcp126, 192.168.201.139@tcp127, 192.168.201.139@tcp128, 192.168.201.139@tcp125, 192.168.201.139@tcp129 [14570.938298] Lustre: lustre-MDT0000: new disk, initializing [14571.012029] Lustre: 464236:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.139@tcp: can't add NID 192.168.201.139@tcp130: rc = -75 [14571.022495] Lustre: 464236:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14571.049414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14571.071209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14571.680117] LNetError: 461639:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.139 [14571.705164] LNetError: 461639:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14571.715499] LNetError: Protocol error connecting to 192.168.201.139@tcp107 at host 192.168.201.139:988: is it running a compatible version of Lustre? [14573.996333] LNetError: 464208:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.139 [14574.005468] LNetError: 464208:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14574.009827] LNetError: Protocol error connecting to 192.168.201.139@tcp122 at host 192.168.201.139:988: is it running a compatible version of Lustre? [14574.022968] LNetError: Skipped 9 previous similar messages [14575.785904] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [14585.312790] LNetError: 461640:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.139 [14585.334572] LNetError: 461640:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14585.347187] LNetError: Protocol error connecting to 192.168.201.139@tcp127 at host 192.168.201.139:988: is it running a compatible version of Lustre? [14587.028377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14587.245641] Lustre: 464196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14587.313971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14587.317866] Lustre: Skipped 1 previous similar message [14587.391838] Lustre: lustre-MDT0001: new disk, initializing [14587.429631] Lustre: 464999:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.139@tcp: can't add NID 192.168.201.139@tcp130: rc = -75 [14587.436468] Lustre: 464999:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14587.540374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14587.588290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14587.601462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14592.179709] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [14595.610875] Lustre: 465387:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.139@tcp: can't add NID 192.168.201.139@tcp130: rc = -75 [14595.616542] Lustre: 465387:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14601.696883] LNetError: 464212:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.139 [14601.718164] LNetError: Protocol error connecting to 192.168.201.139@tcp127 at host 192.168.201.139:988: is it running a compatible version of Lustre? [14602.033779] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14609.000459] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14610.814750] LNetError: 464204:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.139 [14610.835911] LNetError: 464204:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14610.850712] LNetError: Protocol error connecting to 192.168.201.139@tcp121 at host 192.168.201.139:988: is it running a compatible version of Lustre? [14610.870498] LNetError: Skipped 4 previous similar messages [14616.087137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14616.295663] Lustre: lustre-OST0000: new disk, initializing [14616.300454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14616.311817] Lustre: 466173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14616.393725] Lustre: 466178:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.139@tcp: can't add NID 192.168.201.139@tcp130: rc = -75 [14616.404450] Lustre: 466178:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14616.477424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14622.309520] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [14625.360161] Lustre: 466636:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.139@tcp: can't add NID 192.168.201.139@tcp130: rc = -75 [14625.367990] Lustre: 466636:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14625.499811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14625.507572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14625.694131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14630.367602] LNet: 1 local NIs in recovery (showing 1): 192.168.201.139@tcp127 [14631.328459] LNetError: 461639:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.139 [14631.344797] LNetError: 461639:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14631.369292] LNetError: Protocol error connecting to 192.168.201.139@tcp107 at host 192.168.201.139:988: is it running a compatible version of Lustre? [14631.388770] LNetError: Skipped 4 previous similar messages [14633.965285] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14646.240319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14646.241830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14646.256093] LustreError: Skipped 1 previous similar message [14646.256978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14646.283308] Lustre: Skipped 1 previous similar message [14647.788870] Lustre: server umount lustre-OST0000 complete [14653.920620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14653.935434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14653.958809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14653.967793] Lustre: Skipped 1 previous similar message [14656.485123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14656.487340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14656.495886] Lustre: Skipped 1 previous similar message [14658.373722] Lustre: server umount lustre-MDT0000 complete [14663.614182] LustreError: 464188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805880 with bad export cookie 4956418426541504775 [14663.620140] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14663.636790] LustreError: 464188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14664.139488] Lustre: server umount lustre-MDT0001 complete [14672.352432] LNetError: 461639:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.139 [14672.382225] LNetError: 461639:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [14672.400844] LNetError: Protocol error connecting to 192.168.201.139@tcp128 at host 192.168.201.139:988: is it running a compatible version of Lustre? [14672.422349] LNetError: Skipped 8 previous similar messages [14680.252253] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [14683.073691] Key type lgssc unregistered [14683.599274] LNet: 468088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14683.604505] LNetError: 468088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14683.617726] LNet: Removed LNI 192.168.201.139@tcp [14684.639562] LNet: Removed LNI 192.168.201.139@tcp100 [14684.645329] LNetError: 468088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14684.652757] LNetError: 468088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14684.661118] LNet: Removed LNI 192.168.201.139@tcp101 [14685.727799] LNet: Removed LNI 192.168.201.139@tcp102 [14685.735132] LNetError: 468088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14685.741486] LNetError: 468088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14685.746826] LNet: Removed LNI 192.168.201.139@tcp103 [14686.751440] LNet: Removed LNI 192.168.201.139@tcp104 [14687.776287] LNet: Removed LNI 192.168.201.139@tcp105 [14687.778345] LNetError: 468088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14687.801050] LNetError: 468088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14687.828422] LNet: Removed LNI 192.168.201.139@tcp106 [14688.863680] LNet: Removed LNI 192.168.201.139@tcp107 [14689.887382] LNet: Removed LNI 192.168.201.139@tcp108 [14690.914471] LNet: Removed LNI 192.168.201.139@tcp109 [14691.935146] LNet: Removed LNI 192.168.201.139@tcp110 [14691.938975] LNetError: 468088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14691.942485] LNetError: 468088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14691.949652] LNet: Removed LNI 192.168.201.139@tcp111 [14692.962372] LNet: Removed LNI 192.168.201.139@tcp112 [14693.983181] LNet: Removed LNI 192.168.201.139@tcp113 [14695.007094] LNet: Removed LNI 192.168.201.139@tcp114 [14696.031182] LNet: Removed LNI 192.168.201.139@tcp115 [14697.055633] LNet: Removed LNI 192.168.201.139@tcp116 [14698.079448] LNet: Removed LNI 192.168.201.139@tcp117 [14699.103230] LNet: Removed LNI 192.168.201.139@tcp118 [14700.127191] LNet: Removed LNI 192.168.201.139@tcp119 [14700.130255] LNetError: 468088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14700.134748] LNetError: 468088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14700.139359] LNet: Removed LNI 192.168.201.139@tcp120 [14701.154355] LNet: Removed LNI 192.168.201.139@tcp121 [14702.175164] LNet: Removed LNI 192.168.201.139@tcp122 [14703.199400] LNet: Removed LNI 192.168.201.139@tcp123 [14704.223250] LNet: Removed LNI 192.168.201.139@tcp124 [14705.247327] LNet: Removed LNI 192.168.201.139@tcp125 [14706.271494] LNet: Removed LNI 192.168.201.139@tcp126 [14706.277306] LNet: Removed LNI 192.168.201.139@tcp127 [14707.295224] LNet: Removed LNI 192.168.201.139@tcp128 [14708.323246] LNet: Removed LNI 192.168.201.139@tcp129 [14708.325260] LNet: Removed LNI 192.168.201.139@tcp130 [14708.340672] LNet: Removed LNI 192.168.201.139@tcp131 [14708.359397] LNet: Removed LNI 192.168.201.139@tcp132 [14708.373669] LNet: Removed LNI 192.168.201.139@tcp133 [14708.383714] LNet: Removed LNI 192.168.201.139@tcp134 [14708.417656] LNet: Removed LNI 192.168.201.139@tcp135 [14709.164363] Key type .llcrypt unregistered [14709.166865] Key type ._llcrypt unregistered [14733.771856] Key type ._llcrypt registered [14733.774556] Key type .llcrypt registered [14733.919394] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [14749.519677] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [14750.116974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14750.135840] alg: No test for adler32 (adler32-zlib) [14751.389573] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [14751.676081] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [14753.359200] Key type lgssc registered [14754.324940] Lustre: Echo OBD driver; http://www.lustre.org/ [14784.343787] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 00:46:41 (1787806001) [14805.896648] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [14808.887952] Key type lgssc unregistered [14809.100612] LNet: 472177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14809.107785] LNetError: 472177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14809.124606] LNet: Removed LNI 192.168.201.139@tcp [14810.334684] Key type .llcrypt unregistered [14810.336396] Key type ._llcrypt unregistered [14828.805349] Key type ._llcrypt registered [14828.807453] Key type .llcrypt registered [14828.888551] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [14829.935349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14830.123945] alg: No test for adler32 (adler32-zlib) [14831.215388] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [14831.433832] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [14833.079189] Key type lgssc registered [14834.042031] Lustre: Echo OBD driver; http://www.lustre.org/ [14840.475118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14840.487352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14841.638838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14841.659299] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14841.709291] Lustre: lustre-MDT0000: new disk, initializing [14841.793342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14841.811542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14845.536367] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [14856.046309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14856.226353] Lustre: 473392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14856.289347] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14856.295352] Lustre: Skipped 1 previous similar message [14856.373554] Lustre: lustre-MDT0001: new disk, initializing [14856.509842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14856.570699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14856.584935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14862.444866] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [14872.233508] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14877.908882] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14884.861044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14885.105603] Lustre: lustre-OST0000: new disk, initializing [14885.107735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14885.110874] Lustre: 475358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14885.211971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14891.674296] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [14894.111870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14894.120156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14894.207080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14900.884266] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14905.860707] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing lnet_if_list [14912.218184] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14912.250680] LNet: Added LNI 192.168.201.139@tcp10 [8/256/0/180] [14918.613644] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14918.647391] LNet: Added LNI 192.168.201.139@tcp11 [8/256/0/180] [14922.413171] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14922.468485] LNet: Added LNI 192.168.201.139@tcp12 [8/256/0/180] [14927.761978] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14927.807663] LNet: Added LNI 192.168.201.139@tcp13 [8/256/0/180] [14933.025366] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14933.041754] LNet: Added LNI 192.168.201.139@tcp14 [8/256/0/180] [14937.073333] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14937.090686] LNet: Added LNI 192.168.201.139@tcp15 [8/256/0/180] [14942.772140] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14942.794778] LNet: Added LNI 192.168.201.139@tcp16 [8/256/0/180] [14949.390779] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14949.430588] LNet: Added LNI 192.168.201.139@tcp17 [8/256/0/180] [14957.154884] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14957.183040] LNet: Added LNI 192.168.201.139@tcp18 [8/256/0/180] [14964.731621] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14964.754174] LNet: Added LNI 192.168.201.139@tcp19 [8/256/0/180] [14972.156464] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14972.180121] LNet: Added LNI 192.168.201.139@tcp20 [8/256/0/180] [14980.658539] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14980.712822] LNet: Added LNI 192.168.201.139@tcp21 [8/256/0/180] [14989.368121] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14989.452527] LNet: Added LNI 192.168.201.139@tcp22 [8/256/0/180] [14997.349311] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14997.394682] LNet: Added LNI 192.168.201.139@tcp23 [8/256/0/180] [15001.618291] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15001.648556] LNet: Added LNI 192.168.201.139@tcp24 [8/256/0/180] [15007.925805] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15007.954723] LNet: Added LNI 192.168.201.139@tcp25 [8/256/0/180] [15014.848505] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15014.875934] LNet: Added LNI 192.168.201.139@tcp26 [8/256/0/180] [15022.075368] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15022.104702] LNet: Added LNI 192.168.201.139@tcp27 [8/256/0/180] [15030.042810] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15030.093353] LNet: Added LNI 192.168.201.139@tcp28 [8/256/0/180] [15037.774581] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15037.836537] LNet: Added LNI 192.168.201.139@tcp29 [8/256/0/180] [15048.966543] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15049.035530] LNet: Added LNI 192.168.201.139@tcp30 [8/256/0/180] [15055.747167] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15055.781180] LNet: Added LNI 192.168.201.139@tcp31 [8/256/0/180] [15063.032393] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15063.056496] LNet: Added LNI 192.168.201.139@tcp32 [8/256/0/180] [15069.966924] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15069.994514] LNet: Added LNI 192.168.201.139@tcp33 [8/256/0/180] [15076.966192] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15077.000974] LNet: Added LNI 192.168.201.139@tcp34 [8/256/0/180] [15083.523735] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15083.553143] LNet: Added LNI 192.168.201.139@tcp35 [8/256/0/180] [15089.862771] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15089.874548] LNet: Added LNI 192.168.201.139@tcp36 [8/256/0/180] [15098.896263] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15098.952568] LNet: Added LNI 192.168.201.139@tcp37 [8/256/0/180] [15107.318893] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15107.342507] LNet: Added LNI 192.168.201.139@tcp38 [8/256/0/180] [15114.898590] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15114.934859] LNet: Added LNI 192.168.201.139@tcp39 [8/256/0/180] [15122.690990] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15122.705187] LNet: Added LNI 192.168.201.139@tcp40 [8/256/0/180] [15130.644609] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15130.680579] LNet: Added LNI 192.168.201.139@tcp41 [8/256/0/180] [15138.756791] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15138.780080] LNet: Added LNI 192.168.201.139@tcp42 [8/256/0/180] [15146.892674] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15146.945855] LNet: Added LNI 192.168.201.139@tcp43 [8/256/0/180] [15156.942641] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15156.959741] LNet: Added LNI 192.168.201.139@tcp44 [8/256/0/180] [15169.506188] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15169.559634] LNet: Added LNI 192.168.201.139@tcp45 [8/256/0/180] [15179.082260] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15179.106901] LNet: Added LNI 192.168.201.139@tcp46 [8/256/0/180] [15191.080601] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15191.121249] LNet: Added LNI 192.168.201.139@tcp47 [8/256/0/180] [15201.903927] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15201.961824] LNet: Added LNI 192.168.201.139@tcp48 [8/256/0/180] [15214.557614] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15214.606542] LNet: Added LNI 192.168.201.139@tcp49 [8/256/0/180] [15226.716209] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15226.774281] LNet: Added LNI 192.168.201.139@tcp50 [8/256/0/180] [15245.796987] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15245.802519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15245.822841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15245.849277] Lustre: Skipped 2 previous similar messages [15250.914164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15250.920827] Lustre: Skipped 2 previous similar messages [15251.331744] Lustre: server umount lustre-MDT0000 complete [15254.558811] LustreError: 473386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787806471 with bad export cookie 16670290115279133554 [15254.569523] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15254.578515] LustreError: 473386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15254.863593] Lustre: server umount lustre-MDT0001 complete [15272.424718] Lustre: 472663:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787806473/real 1787806473] req@00000000f98857ed x1874650514774272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787806489 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15272.501871] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15272.530306] Lustre: Skipped 2 previous similar messages [15275.167724] Lustre: 472664:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787806476/real 1787806476] req@00000000303acb54 x1874650514774400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787806492 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15275.933880] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [15277.548920] Lustre: 472661:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787806478/real 1787806478] req@0000000064dd1d0c x1874650514774528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787806494 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15280.864378] Lustre: 472661:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787806481/real 1787806481] req@00000000ba5631ef x1874650514774784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787806497 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15285.739428] Lustre: 472663:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787806486/real 1787806486] req@0000000069b521dd x1874650514774912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787806502 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15286.533734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15299.300398] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe758c4896feb2590 [15299.318245] Lustre: MGC192.168.201.139@tcp: Connection restored to 0@lo (at 0@lo) [15299.689281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15301.731468] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15305.283767] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [15308.258840] LustreError: 475352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15308.284329] LustreError: 475352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15313.382125] LustreError: 475353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15315.344844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15315.949215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15318.001184] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15323.060904] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [15335.198953] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15344.394659] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15364.073644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15364.122666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15364.144889] Lustre: Skipped 1 previous similar message [15369.197967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15369.207049] Lustre: Skipped 1 previous similar message [15374.307517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15374.318197] Lustre: Skipped 1 previous similar message [15378.399947] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15378.621588] Lustre: server umount lustre-OST0000 complete [15384.559862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15384.569073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15384.571038] Lustre: Skipped 2 previous similar messages [15384.579987] Lustre: Skipped 1 previous similar message [15389.667919] LustreError: 499612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15389.718539] Lustre: server umount lustre-MDT0000 complete [15393.532244] LustreError: 474187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787806610 with bad export cookie 16670290115279136144 [15393.536240] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15393.547019] LustreError: 474187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15393.857072] Lustre: server umount lustre-MDT0001 complete [15407.968293] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [15411.552053] Key type lgssc unregistered [15412.260234] LNet: 502396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15412.266355] LNetError: 502396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15412.268871] LNet: Removed LNI 192.168.201.139@tcp [15412.271967] LNet: Removed LNI 192.168.201.139@tcp10 [15412.278468] LNet: Removed LNI 192.168.201.139@tcp11 [15412.282103] LNet: Removed LNI 192.168.201.139@tcp12 [15412.283914] LNet: Removed LNI 192.168.201.139@tcp13 [15412.288159] LNet: Removed LNI 192.168.201.139@tcp14 [15412.292816] LNet: Removed LNI 192.168.201.139@tcp15 [15412.296978] LNet: Removed LNI 192.168.201.139@tcp16 [15412.300283] LNet: Removed LNI 192.168.201.139@tcp17 [15412.302770] LNet: Removed LNI 192.168.201.139@tcp18 [15412.304614] LNet: Removed LNI 192.168.201.139@tcp19 [15412.308738] LNet: Removed LNI 192.168.201.139@tcp20 [15412.317470] LNet: Removed LNI 192.168.201.139@tcp21 [15412.328667] LNet: Removed LNI 192.168.201.139@tcp22 [15412.344877] LNet: Removed LNI 192.168.201.139@tcp23 [15412.358719] LNet: Removed LNI 192.168.201.139@tcp24 [15412.377157] LNet: Removed LNI 192.168.201.139@tcp25 [15412.381242] LNet: Removed LNI 192.168.201.139@tcp26 [15412.393799] LNet: Removed LNI 192.168.201.139@tcp27 [15412.406439] LNet: Removed LNI 192.168.201.139@tcp28 [15412.417899] LNet: Removed LNI 192.168.201.139@tcp29 [15412.431619] LNet: Removed LNI 192.168.201.139@tcp30 [15412.439214] LNet: Removed LNI 192.168.201.139@tcp31 [15412.445937] LNet: Removed LNI 192.168.201.139@tcp32 [15412.450806] LNet: Removed LNI 192.168.201.139@tcp33 [15412.465210] LNet: Removed LNI 192.168.201.139@tcp34 [15412.469534] LNet: Removed LNI 192.168.201.139@tcp35 [15412.476744] LNet: Removed LNI 192.168.201.139@tcp36 [15412.485634] LNet: Removed LNI 192.168.201.139@tcp37 [15412.494593] LNet: Removed LNI 192.168.201.139@tcp38 [15412.502418] LNet: Removed LNI 192.168.201.139@tcp39 [15412.504354] LNet: Removed LNI 192.168.201.139@tcp40 [15412.514465] LNet: Removed LNI 192.168.201.139@tcp41 [15412.530470] LNet: Removed LNI 192.168.201.139@tcp42 [15412.540901] LNet: Removed LNI 192.168.201.139@tcp43 [15412.553302] LNet: Removed LNI 192.168.201.139@tcp44 [15412.555079] LNet: Removed LNI 192.168.201.139@tcp45 [15412.562923] LNet: Removed LNI 192.168.201.139@tcp46 [15412.572744] LNet: Removed LNI 192.168.201.139@tcp47 [15412.578827] LNet: Removed LNI 192.168.201.139@tcp48 [15412.583147] LNet: Removed LNI 192.168.201.139@tcp49 [15412.594416] LNet: Removed LNI 192.168.201.139@tcp50 [15414.067685] Key type .llcrypt unregistered [15414.069570] Key type ._llcrypt unregistered [15439.996815] Key type ._llcrypt registered [15439.999949] Key type .llcrypt registered [15440.153456] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [15461.830255] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [15464.206260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15464.281524] alg: No test for adler32 (adler32-zlib) [15465.415581] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [15465.562529] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [15467.207183] Key type lgssc registered [15468.708682] Lustre: Echo OBD driver; http://www.lustre.org/ [15513.627772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15513.678273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15515.053205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15515.149714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15515.326196] Lustre: lustre-MDT0000: new disk, initializing [15515.496113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15515.547179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15523.051610] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [15539.717342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15539.859172] Lustre: 505600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15539.927313] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15539.937043] Lustre: Skipped 1 previous similar message [15540.054779] Lustre: lustre-MDT0001: new disk, initializing [15540.183231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15540.296667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15540.317668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15546.737916] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [15557.955109] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15564.714891] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15573.788774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15573.989434] Lustre: *** cfs_fail_loc=90f, val=0*** [15574.046593] Lustre: lustre-OST0000: new disk, initializing [15574.050429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15574.055766] Lustre: 507623:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15574.151339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15579.771266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15579.804368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15579.974788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15581.739757] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [15594.578884] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15605.729867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15605.746396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15605.784033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15610.873385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15610.892933] Lustre: Skipped 2 previous similar messages [15611.567395] Lustre: server umount lustre-OST0000 complete [15616.993134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15617.005240] LustreError: Skipped 1 previous similar message [15617.012393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15617.040729] Lustre: Skipped 1 previous similar message [15617.061327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15621.088377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15621.090696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15621.125183] Lustre: Skipped 1 previous similar message [15622.557569] Lustre: server umount lustre-MDT0000 complete [15627.705934] LustreError: 505593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787806844 with bad export cookie 4948975395804483104 [15627.710529] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15627.723426] LustreError: 505593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15628.168592] Lustre: server umount lustre-MDT0001 complete [15643.940986] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [15647.522703] Key type lgssc unregistered [15647.753845] LNet: 509546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15647.766394] LNetError: 509546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15647.798068] LNet: Removed LNI 192.168.201.139@tcp [15648.852321] Key type .llcrypt unregistered [15648.853715] Key type ._llcrypt unregistered [15677.864819] Key type ._llcrypt registered [15677.871881] Key type .llcrypt registered [15678.214239] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [15680.215109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15680.478153] alg: No test for adler32 (adler32-zlib) [15681.888872] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [15682.134210] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [15683.904188] Key type lgssc registered [15685.327745] Lustre: Echo OBD driver; http://www.lustre.org/ [15694.574620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15694.593667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15696.345085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15701.138470] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [15706.600710] LustreError: 510818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15706.637488] LustreError: 510818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15711.728328] LustreError: 510818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15712.419302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15712.803476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15718.589425] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [15727.847864] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15734.506392] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15743.865724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15744.337647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15754.875993] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [15765.482817] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15778.354198] loop0: detected capacity change from 0 to 204800000 [15778.552730] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15778.556864] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15778.567146] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15786.142986] loop0: detected capacity change from 0 to 204800000 [15786.408195] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15786.417914] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15786.448457] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15793.509870] loop0: detected capacity change from 0 to 204800000 [15801.814537] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15802.112578] Lustre: fs15246-OST0000: new disk, initializing [15802.122759] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15802.130839] Lustre: 514415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15802.229205] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15812.177524] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [15826.940591] loop1: detected capacity change from 0 to 204800000 [15836.190315] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15836.426207] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15836.548386] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15836.666224] Lustre: fs15246-MDT0000: new disk, initializing [15836.969975] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15837.070234] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15837.228523] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15838.304888] Lustre: 515540:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000941edc78 x1874651406982656/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787807071 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15842.842643] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [15843.434766] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15843.445079] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15843.500773] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15873.518036] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15873.535498] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15875.857200] Lustre: server umount fs15246-MDT0000 complete [15878.641953] LustreError: 513261:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15883.775551] LustreError: 510813:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15887.963931] Lustre: server umount fs15246-OST0000 complete [15904.233814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15904.243810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15904.263906] Lustre: Skipped 1 previous similar message [15904.274062] Lustre: Skipped 1 previous similar message [15909.174474] Lustre: server umount lustre-OST0000 complete [15914.471011] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15914.491307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15919.584685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15919.599340] Lustre: Skipped 2 previous similar messages [15920.167431] Lustre: server umount lustre-MDT0000 complete [15924.673971] LustreError: 510801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787807141 with bad export cookie 7615415647300646750 [15924.690989] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15924.698257] LustreError: 510801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15925.286073] Lustre: server umount lustre-MDT0001 complete [15938.838564] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [15942.237269] Key type lgssc unregistered [15942.654726] LNet: 518643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15942.663915] LNetError: 518643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15942.687686] LNet: Removed LNI 192.168.201.139@tcp [15943.862347] Key type .llcrypt unregistered [15943.867864] Key type ._llcrypt unregistered [15965.642862] Key type ._llcrypt registered [15965.644968] Key type .llcrypt registered [15965.770266] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [15966.883354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15967.010701] alg: No test for adler32 (adler32-zlib) [15968.049360] Lustre: Lustre: Build Version: 2.17.57_50_g504224c [15968.218349] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [15969.864100] Key type lgssc registered [15971.190154] Lustre: Echo OBD driver; http://www.lustre.org/ [16020.301894] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [16029.803685] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [16069.732871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16069.781758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16071.272387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16071.317459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16071.450510] Lustre: lustre-MDT0000: new disk, initializing [16071.588114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16071.641261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16077.460881] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [16090.778300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16090.974659] Lustre: 523114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16091.031902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16091.040846] Lustre: Skipped 1 previous similar message [16091.178753] Lustre: lustre-MDT0001: new disk, initializing [16091.265374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16091.309049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16091.315276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16095.320994] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [16107.689820] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16117.804173] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16130.427916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16130.683914] Lustre: lustre-OST0000: new disk, initializing [16130.695592] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16130.713418] Lustre: 525085:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16130.901012] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16136.291572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16136.306168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16136.583197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16138.887094] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [16150.367942] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16155.884664] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16156.110225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16161.509160] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16161.691376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16167.392796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16167.395118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16167.406146] LustreError: Skipped 1 previous similar message [16167.411873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16167.422159] Lustre: Skipped 1 previous similar message [16167.441260] Lustre: Skipped 1 previous similar message [16169.882745] Lustre: server umount lustre-OST0000 complete [16177.633118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16177.644238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16177.661793] Lustre: Skipped 1 previous similar message [16181.077262] Lustre: server umount lustre-MDT0000 complete [16186.270836] LustreError: 523909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787807403 with bad export cookie 2066387877030050412 [16186.281325] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16186.290735] LustreError: 523909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16186.843438] Lustre: server umount lustre-MDT0001 complete [16217.023545] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [16229.556342] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [16266.836450] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16012 sec ========= 01:11:23 (1787807483) [16269.192312] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:11:25 (1787807485) === [16273.505980] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:11:29 (1787807489) === [16321.273716] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [16325.493618] Key type lgssc unregistered [16325.728978] LNet: 531740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16325.739473] LNetError: 531740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16326.758983] LNet: Removed LNI 192.168.201.139@tcp [16327.630555] Key type .llcrypt unregistered [16327.632916] Key type ._llcrypt unregistered