[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 481114146 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003193] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009020] pid_max: default: 32768 minimum: 301 [ 0.010159] LSM: Security Framework initializing [ 0.011068] Yama: becoming mindful. [ 0.012048] SELinux: Initializing. [ 0.013088] *** VALIDATE selinux *** [ 0.022396] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027480] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028174] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029129] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031139] *** VALIDATE tmpfs *** [ 0.033018] *** VALIDATE proc *** [ 0.035023] *** VALIDATE cgroup *** [ 0.036017] *** VALIDATE cgroup2 *** [ 0.038305] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040083] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043029] Spectre V2 : User space: Vulnerable [ 0.044014] Speculative Store Bypass: Vulnerable [ 0.047972] debug: unmapping init [mem 0xffffffff95459000-0xffffffff95460fff] [ 0.051241] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.052812] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.053030] ... version: 2 [ 0.054020] ... bit width: 48 [ 0.055019] ... generic registers: 4 [ 0.056023] ... value mask: 0000ffffffffffff [ 0.057023] ... max period: 00007fffffffffff [ 0.058022] ... fixed-purpose events: 3 [ 0.059017] ... event mask: 000000070000000f [ 0.060349] rcu: Hierarchical SRCU implementation. [ 0.062547] smp: Bringing up secondary CPUs ... [ 0.063741] x86: Booting SMP configuration: [ 0.064042] .... node #0, CPUs: #1 #2 #3 [ 0.068018] smp: Brought up 1 node, 4 CPUs [ 0.070025] smpboot: Max logical packages: 1 [ 0.071024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.240899] node 0 deferred pages initialised in 167ms [ 0.243288] devtmpfs: initialized [ 0.245537] x86/mm: Memory block size: 128MB [ 0.249237] gcov: version magic: 0x41383552 [ 0.252497] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.257119] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.260397] pinctrl core: initialized pinctrl subsystem [ 0.263251] [ 0.263952] ************************************************************* [ 0.266014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.269017] ** ** [ 0.272020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.275015] ** ** [ 0.277012] ** This means that this kernel is built to expose internal ** [ 0.279014] ** IOMMU data structures, which may compromise security on ** [ 0.281015] ** your system. ** [ 0.284017] ** ** [ 0.286014] ** If you see this message and you are not debugging the ** [ 0.289021] ** kernel, report this immediately to your vendor! ** [ 0.292017] ** ** [ 0.294014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.297019] ************************************************************* [ 0.300762] NET: Registered protocol family 16 [ 0.302559] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.306077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.308074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.313020] cpuidle: using governor menu [ 0.314974] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.319095] PCI: Using configuration type 1 for base access [ 0.322130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.332142] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.334027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.338160] cryptd: max_cpu_qlen set to 1000 [ 0.341252] ACPI: Added _OSI(Module Device) [ 0.343022] ACPI: Added _OSI(Processor Device) [ 0.344014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.345014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.349732] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.355339] ACPI: Interpreter enabled [ 0.357077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.359031] ACPI: Using IOAPIC for interrupt routing [ 0.361123] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.363531] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.374089] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.377056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.380039] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.382107] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.386073] acpiphp: Slot [2] registered [ 0.387169] acpiphp: Slot [5] registered [ 0.388151] acpiphp: Slot [6] registered [ 0.390192] acpiphp: Slot [7] registered [ 0.391195] acpiphp: Slot [8] registered [ 0.393100] acpiphp: Slot [9] registered [ 0.394100] acpiphp: Slot [10] registered [ 0.395120] acpiphp: Slot [3] registered [ 0.396102] acpiphp: Slot [4] registered [ 0.397105] acpiphp: Slot [11] registered [ 0.398124] acpiphp: Slot [12] registered [ 0.400100] acpiphp: Slot [13] registered [ 0.401097] acpiphp: Slot [14] registered [ 0.402102] acpiphp: Slot [15] registered [ 0.403088] acpiphp: Slot [16] registered [ 0.404092] acpiphp: Slot [17] registered [ 0.404996] acpiphp: Slot [18] registered [ 0.406110] acpiphp: Slot [19] registered [ 0.407105] acpiphp: Slot [20] registered [ 0.409144] acpiphp: Slot [21] registered [ 0.410126] acpiphp: Slot [22] registered [ 0.412110] acpiphp: Slot [23] registered [ 0.413115] acpiphp: Slot [24] registered [ 0.415094] acpiphp: Slot [25] registered [ 0.417135] acpiphp: Slot [26] registered [ 0.419230] acpiphp: Slot [27] registered [ 0.420143] acpiphp: Slot [28] registered [ 0.422144] acpiphp: Slot [29] registered [ 0.424123] acpiphp: Slot [30] registered [ 0.425153] acpiphp: Slot [31] registered [ 0.427083] PCI host bridge to bus 0000:00 [ 0.429041] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.432031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.434026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.437031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.439029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.442031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.444196] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.447121] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.451251] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.462017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.466490] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.469022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.471017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.473022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.476024] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.478826] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.481059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.483790] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.487810] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.499016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.504020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.509794] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.519020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.530022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.548021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.563663] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.571018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.577016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.591031] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.605000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.612020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.619018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.635018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.649997] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.655017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.661013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.677017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.686866] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.694018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.703069] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.719016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.726984] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.737016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.750017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.772019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.783122] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.785619] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.787336] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.789342] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.791228] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.795579] iommu: Default domain type: Passthrough [ 0.797512] SCSI subsystem initialized [ 0.798142] ACPI: bus type USB registered [ 0.801136] usbcore: registered new interface driver usbfs [ 0.803102] usbcore: registered new interface driver hub [ 0.805065] usbcore: registered new device driver usb [ 0.806155] pps_core: LinuxPPS API ver. 1 registered [ 0.807010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.811077] PTP clock support registered [ 0.813050] EDAC MC: Ver: 3.0.0 [ 0.814365] PCI: Using ACPI for IRQ routing [ 0.815477] NetLabel: Initializing [ 0.816000] NetLabel: domain hash size = 128 [ 0.818017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.820084] NetLabel: unlabeled traffic allowed by default [ 0.822153] vgaarb: loaded [ 0.824266] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.826012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.831364] clocksource: Switched to clocksource kvm-clock [ 0.945192] VFS: Disk quotas dquot_6.6.0 [ 0.947382] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.950541] *** VALIDATE ramfs *** [ 0.952178] *** VALIDATE hugetlbfs *** [ 0.954339] pnp: PnP ACPI init [ 0.956800] pnp: PnP ACPI: found 6 devices [ 0.977136] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.980029] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.981426] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.982931] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.984970] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.987756] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.990055] NET: Registered protocol family 2 [ 0.992434] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.996809] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.000313] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.005131] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.010661] TCP: Hash tables configured (established 65536 bind 65536) [ 1.013847] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.016888] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.019875] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.022938] NET: Registered protocol family 1 [ 1.026557] RPC: Registered named UNIX socket transport module. [ 1.028674] RPC: Registered udp transport module. [ 1.031231] RPC: Registered tcp transport module. [ 1.032984] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.035802] NET: Registered protocol family 44 [ 1.037677] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.040129] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.042575] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.044841] PCI: CLS 0 bytes, default 64 [ 1.046635] Unpacking initramfs... [ 2.444179] debug: unmapping init [mem 0xffff9eb03cc54000-0xffff9eb03ffbffff] [ 2.447605] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.449380] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.451718] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.951885] Initialise system trusted keyrings [ 2.953804] Key type blacklist registered [ 2.956267] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.964072] zbud: loaded [ 2.967382] *** VALIDATE nfs *** [ 2.968948] *** VALIDATE nfs4 *** [ 2.971210] pstore: using deflate compression [ 2.980716] Platform Keyring initialized [ 3.101910] NET: Registered protocol family 38 [ 3.103757] Key type asymmetric registered [ 3.105290] Asymmetric key parser 'x509' registered [ 3.107223] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.112875] io scheduler mq-deadline registered [ 3.114702] io scheduler kyber registered [ 3.116591] io scheduler bfq registered [ 3.118769] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.122119] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.125153] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.128671] ACPI: Power Button [PWRF] [ 3.135274] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.141895] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.157312] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.172731] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.206628] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.235907] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.267212] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.272446] Non-volatile memory driver v1.3 [ 3.274381] Linux agpgart interface v0.103 [ 3.306768] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.311324] vda: detected capacity change from 0 to 74698752 [ 3.333674] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.337018] vdb: detected capacity change from 0 to 1073741824 [ 3.350931] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.354395] vdc: detected capacity change from 0 to 2621440000 [ 3.371445] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.374165] vdd: detected capacity change from 0 to 2621440000 [ 3.389338] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.392488] vde: detected capacity change from 0 to 4294967296 [ 3.408325] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.411741] vdf: detected capacity change from 0 to 4294967296 [ 3.420153] libphy: Fixed MDIO Bus: probed [ 3.432393] usbcore: registered new interface driver usbserial_generic [ 3.434547] usbserial: USB Serial support registered for generic [ 3.436499] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.440052] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.441613] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.443718] mousedev: PS/2 mouse device common for all mice [ 3.446200] rtc_cmos 00:05: RTC can wake from S4 [ 3.448739] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.452950] rtc_cmos 00:05: registered as rtc0 [ 3.455557] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.456155] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.459226] intel_pstate: CPU model not supported [ 3.464110] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.469434] hid: raw HID events driver (C) Jiri Kosina [ 3.471724] usbcore: registered new interface driver usbhid [ 3.474028] usbhid: USB HID core driver [ 3.475784] drop_monitor: Initializing network drop monitor service [ 3.478358] Initializing XFRM netlink socket [ 3.480575] NET: Registered protocol family 10 [ 3.483438] Segment Routing with IPv6 [ 3.484749] NET: Registered protocol family 17 [ 3.486863] mpls_gso: MPLS GSO support [ 3.492736] RAS: Correctable Errors collector initialized. [ 3.494900] AVX version of gcm_enc/dec engaged. [ 3.496957] AES CTR mode by8 optimization enabled [ 3.571706] sched_clock: Marking stable (3571682646, 0)->(4597061680, -1025379034) [ 3.574686] registered taskstats version 1 [ 3.576842] Loading compiled-in X.509 certificates [ 3.579222] zswap: loaded using pool lzo/zbud [ 3.607565] Key type big_key registered [ 3.618864] Key type encrypted registered [ 3.620718] ima: No TPM chip found, activating TPM-bypass! [ 3.623115] ima: Allocated hash algorithm: sha1 [ 3.624918] ima: No architecture policies found [ 3.626824] evm: Initialising EVM extended attributes: [ 3.628944] evm: security.selinux [ 3.630411] evm: security.ima [ 3.631612] evm: security.capability [ 3.632946] evm: HMAC attrs: 0x1 [ 3.635336] rtc_cmos 00:05: setting system clock to 2026-08-15 16:53:54 UTC (1786812834) [ 3.641830] debug: unmapping init [mem 0xffffffff96403000-0xffffffff965fffff] [ 3.645882] debug: unmapping init [mem 0xffffffff95182000-0xffffffff95458fff] [ 3.655088] Write protecting the kernel read-only data: 28672k [ 3.658222] debug: unmapping init [mem 0xffffffff93803000-0xffffffff939fffff] [ 3.661012] debug: unmapping init [mem 0xffffffff94114000-0xffffffff941fffff] [ 3.698067] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.707362] systemd[1]: Detected virtualization kvm. [ 3.709451] systemd[1]: Detected architecture x86-64. [ 3.711374] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.737888] systemd[1]: No hostname configured. [ 3.739872] systemd[1]: Set hostname to . [ 3.742244] random: systemd: uninitialized urandom read (16 bytes read) [ 3.744962] systemd[1]: Initializing machine ID from random generator. [ 3.777768] random: ln: uninitialized urandom read (6 bytes read) [ 3.875667] random: systemd: uninitialized urandom read (16 bytes read) [ 3.877854] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.883056] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.890301] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.536254] device-mapper: uevent: version 1.0.3 [ 4.538706] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK [ 5.144550] random: fast init done ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.317639] virtio_net virtio0 ens2: renamed from eth0 [ 5.389627] scsi host0: ata_piix [ 5.412746] scsi host1: ata_piix [ 5.414079] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.417235] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.197681] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.138578] random: crng init done [ 10.139996] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.679856] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.917774] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.184255] SELinux: Disabled at runtime. [ 12.247583] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.256471] systemd[1]: Detected virtualization kvm. [ 12.258987] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.833251] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.837491] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.842323] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.848338] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.851844] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.859432] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.867251] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ 12.978775] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.358230] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.722609] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.786861] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.952305] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.971543] EDAC sbridge: Ver: 1.1.2 [ 15.584386] Key type dns_resolver registered [ 15.900573] NFS: Registering the id_resolver key type [ 15.902861] Key type id_resolver registered [ 15.904523] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg627-server login: [ 51.186788] libcfs: loading out-of-tree module taints kernel. [ 51.251986] Key type ._llcrypt registered [ 51.257118] Key type .llcrypt registered [ 51.422923] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 67.551018] hrtimer: interrupt took 7007648 ns [ 68.975270] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 70.348634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.362111] alg: No test for adler32 (adler32-zlib) [ 71.741969] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 72.438046] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 74.129634] Key type lgssc registered [ 76.493902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.545647] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 136.875408] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 151.968146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 152.044402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.274113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 153.297519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 153.424545] Lustre: lustre-MDT0000: new disk, initializing [ 153.484470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 153.499088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 159.044829] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.845843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.908718] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 172.956968] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 172.966983] Lustre: Skipped 1 previous similar message [ 173.039568] Lustre: lustre-MDT0001: new disk, initializing [ 173.097952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 173.145339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 173.163481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.505523] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.142208] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 196.419062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.808694] Lustre: lustre-OST0000: new disk, initializing [ 196.811889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 196.817075] Lustre: 8457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.883478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.746254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 201.767533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 201.851462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.038401] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.960268] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.137691] Lustre: lustre-OST0001: new disk, initializing [ 220.142673] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 220.157459] Lustre: 9531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.230407] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.874559] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.877780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 228.891530] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 228.950057] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 238.350071] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 243.420340] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 249.623208] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing check_logdir /tmp/testlogs/ [ 254.863763] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing yml_node [ 261.131601] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 264.973705] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 268.511521] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 270.889267] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 12:58:19 EDT 2026 [ 286.724824] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 288.094233] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 305.633850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.636849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 305.646631] Lustre: Skipped 3 previous similar messages [ 305.650412] Lustre: Skipped 2 previous similar messages [ 309.136752] Lustre: server umount lustre-MDT0000 complete [ 311.903412] LustreError: 7476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813142 with bad export cookie 13723265750659253021 [ 311.905707] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 311.918805] LustreError: 7476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 315.873125] LustreError: 6525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 315.874140] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.874730] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 315.898408] LustreError: 6525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 315.930740] Lustre: Skipped 1 previous similar message [ 316.897801] LustreError: 6524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.898279] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.921925] Lustre: Skipped 2 previous similar messages [ 318.249233] Lustre: server umount lustre-MDT0001 complete [ 321.925074] Lustre: server umount lustre-OST0000 complete [ 325.612395] Lustre: server umount lustre-OST0001 complete [ 332.432540] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 341.166129] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 376.729659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 377.081289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 377.133616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 377.230954] Lustre: lustre-MDT0000: new disk, initializing [ 377.381926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 377.403992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 382.454882] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 392.350771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 392.569811] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 392.692292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 392.703373] Lustre: Skipped 1 previous similar message [ 392.769619] Lustre: lustre-MDT0001: new disk, initializing [ 392.943778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 393.023194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 393.049795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 396.851810] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 406.107850] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 412.755767] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 421.586460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 422.085216] Lustre: lustre-OST0000: new disk, initializing [ 422.089688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 422.096939] Lustre: 17050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 422.295409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.021544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 424.037499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 424.233845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 431.748069] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 443.771639] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 449.360678] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 449.475684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 453.497972] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 453.697267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 455.139399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 455.144849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.151806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 455.163091] Lustre: Skipped 1 previous similar message [ 455.171099] Lustre: Skipped 1 previous similar message [ 460.258085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 460.263301] Lustre: Skipped 1 previous similar message [ 461.359317] Lustre: server umount lustre-OST0000 complete [ 464.866265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 464.873330] LustreError: Skipped 1 previous similar message [ 464.878112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.497791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 470.512062] Lustre: Skipped 4 previous similar messages [ 470.986266] Lustre: server umount lustre-MDT0000 complete [ 474.533801] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813305 with bad export cookie 13723265750659254554 [ 474.544962] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.846785] Lustre: server umount lustre-MDT0001 complete [ 478.564510] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:01:48 (1786813308) [ 481.445335] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 483.541893] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:01:52 (1786813312) [ 495.446401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.509396] LustreError: 19474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.548583] LustreError: 19474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 496.666305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.732528] LustreError: 19475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.159558] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 506.854368] LustreError: 19474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.163884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.361338] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 525.502245] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 530.844561] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 537.926104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 545.563233] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 554.320770] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 567.275524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 567.281858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.298646] Lustre: Skipped 1 previous similar message [ 567.302060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 571.754747] Lustre: server umount lustre-OST0000 complete [ 578.532113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.551467] Lustre: Skipped 1 previous similar message [ 590.304479] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 590.585363] Lustre: server umount lustre-MDT0000 complete [ 593.892923] LustreError: 19475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.913304] LustreError: 19475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 594.523603] LustreError: 19455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813425 with bad export cookie 13723265750659255436 [ 594.528485] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 594.542096] LustreError: 19455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 594.845573] Lustre: server umount lustre-MDT0001 complete [ 605.038559] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 607.796430] Key type lgssc unregistered [ 608.134265] LNet: 23010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.146185] LNetError: 23010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 608.174166] LNet: Removed LNI 192.168.206.127@tcp [ 609.256159] Key type .llcrypt unregistered [ 609.261475] Key type ._llcrypt unregistered [ 632.386203] Key type ._llcrypt registered [ 632.390484] Key type .llcrypt registered [ 632.495938] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 633.500937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 633.518746] alg: No test for adler32 (adler32-zlib) [ 634.625205] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 634.806426] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 636.464163] Key type lgssc registered [ 637.654632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 647.518311] loop: module loaded [ 647.584138] loop0: detected capacity change from 0 to 204800000 [ 647.774724] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 647.788201] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 648.002093] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 654.617192] loop0: detected capacity change from 0 to 204800000 [ 655.000399] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 655.009681] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 655.021686] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 662.432717] loop0: detected capacity change from 0 to 204800000 [ 670.583418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 670.615100] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 671.993399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 672.021479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 672.102197] Lustre: lustre-MDT0000: new disk, initializing [ 672.157039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 672.177628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 678.982585] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 689.153120] loop1: detected capacity change from 0 to 204800000 [ 696.669088] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 697.098668] Lustre: lustre-OST07c6: new disk, initializing [ 697.103543] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 697.107360] Lustre: Skipped 1 previous similar message [ 697.113514] Lustre: 26200:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 697.192805] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 704.084082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 704.096629] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 704.139528] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 704.535721] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 711.085810] Lustre: 25185:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 711.096048] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 711.100774] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 711.110594] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 711.116617] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 711.123884] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 725.790187] Lustre: 25184:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 725.808042] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 725.816548] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 725.824907] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 725.830186] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 725.834486] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 731.350173] Lustre: 25186:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 731.359392] Lustre: 25186:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 731.364339] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 731.369968] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.376168] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 731.381432] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.387092] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 731.393465] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.399061] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 731.404176] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.409770] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 731.415938] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.619692] Lustre: 25184:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 734.632267] Lustre: 25184:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 734.649088] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 734.669780] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.679730] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 734.697329] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.711889] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 734.724371] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.739692] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 734.750284] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.758950] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 734.768506] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.220177] Lustre: 25185:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 740.237922] Lustre: 25185:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 740.249800] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 740.270248] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.283610] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 740.296039] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.307418] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 740.316836] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.330572] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 740.336818] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.342510] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 740.347952] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.502818] Lustre: 25184:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 748.513834] Lustre: 25184:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 748.522269] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 748.533028] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 748.542556] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 748.551621] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 748.559190] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 748.564240] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 748.570134] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 748.576399] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 748.582081] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 748.590230] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 765.126562] Lustre: 25184:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 765.152061] Lustre: 25184:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 7 previous similar messages [ 765.164124] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 765.173985] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 765.186086] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 765.204699] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 765.212372] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 765.220200] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 765.236463] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 765.242471] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 765.252876] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 765.260186] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 806.378785] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 806.382763] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 806.417139] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 810.206194] Lustre: server umount lustre-OST07c6 complete [ 820.973626] Lustre: server umount lustre-MDT0000 complete [ 837.102109] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 840.169799] Key type lgssc unregistered [ 840.450626] LNet: 28764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.471336] LNetError: 28764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.493703] LNet: Removed LNI 192.168.206.127@tcp [ 841.261170] Key type .llcrypt unregistered [ 841.266179] Key type ._llcrypt unregistered [ 858.018564] Key type ._llcrypt registered [ 858.022070] Key type .llcrypt registered [ 858.115377] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 859.009729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 859.044172] alg: No test for adler32 (adler32-zlib) [ 860.068932] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 860.312926] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 861.952218] Key type lgssc registered [ 862.934866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 869.892528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 869.920436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.398603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 876.144857] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 881.634549] LustreError: 30014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 881.650606] LustreError: 30014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 885.082688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 885.448464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 889.207148] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 897.240090] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 905.125738] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 913.208682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 913.557275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.650459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 920.670534] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 930.936624] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 951.777874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 951.785216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.795041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 951.799863] Lustre: Skipped 1 previous similar message [ 951.808253] Lustre: Skipped 1 previous similar message [ 954.348993] Lustre: server umount lustre-OST0000 complete [ 962.018658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.023287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 962.036187] Lustre: Skipped 1 previous similar message [ 962.043765] Lustre: Skipped 1 previous similar message [ 964.642798] Lustre: server umount lustre-MDT0000 complete [ 967.137295] LustreError: 30010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 967.183730] LustreError: 30010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 968.626889] LustreError: 29996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813799 with bad export cookie 1123259034910700999 [ 968.646276] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 968.647844] LustreError: 29996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 969.095248] Lustre: server umount lustre-MDT0001 complete [ 980.474603] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 983.300725] Key type lgssc unregistered [ 983.581447] LNet: 33758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.585522] LNetError: 33758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.610879] LNet: Removed LNI 192.168.206.127@tcp [ 984.512986] Key type .llcrypt unregistered [ 984.515874] Key type ._llcrypt unregistered [ 1002.427363] Key type ._llcrypt registered [ 1002.430638] Key type .llcrypt registered [ 1002.562946] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1003.241761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1003.326880] alg: No test for adler32 (adler32-zlib) [ 1004.457172] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 1004.645737] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 1006.288319] Key type lgssc registered [ 1007.540608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1016.990434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1017.004278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.655896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.264848] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1029.101386] LustreError: 35026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1029.110577] LustreError: 35026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1032.669263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.951362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1037.433385] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1044.741729] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1050.118629] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1056.511659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1056.958624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1062.811908] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1064.128519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1069.339895] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1078.757614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1078.773717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.786048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1079.779502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1079.779990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1079.780478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.815312] Lustre: Skipped 1 previous similar message [ 1084.648860] Lustre: server umount lustre-OST0000 complete [ 1089.505163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1089.514972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.531267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1095.137837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1095.144349] Lustre: Skipped 3 previous similar messages [ 1100.263360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1100.275022] Lustre: Skipped 1 previous similar message [ 1102.304370] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1102.481453] Lustre: server umount lustre-MDT0000 complete [ 1104.355071] LustreError: 35021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1104.389143] LustreError: 35021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1105.981285] LustreError: 36869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813936 with bad export cookie 12451771186572892239 [ 1105.984364] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1105.990108] LustreError: 36869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1106.394920] Lustre: server umount lustre-MDT0001 complete [ 1115.841358] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 1118.051941] Key type lgssc unregistered [ 1118.297045] LNet: 38563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.311244] LNetError: 38563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.335266] LNet: Removed LNI 192.168.206.127@tcp [ 1119.158569] Key type .llcrypt unregistered [ 1119.161522] Key type ._llcrypt unregistered [ 1137.923446] Key type ._llcrypt registered [ 1137.926463] Key type .llcrypt registered [ 1138.014662] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1138.726192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1138.808194] alg: No test for adler32 (adler32-zlib) [ 1139.759615] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 1139.933746] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 1141.592499] Key type lgssc registered [ 1142.577679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1149.009538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1149.035381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.486436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.017745] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1160.450130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.723237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1164.256740] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1170.028769] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1175.059103] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1181.269087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1181.592161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1186.477397] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1191.828595] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1195.950749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1203.834091] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.27@tcp (stopping) [ 1206.245044] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.251548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1206.262619] Lustre: Skipped 2 previous similar messages [ 1206.269949] Lustre: Skipped 1 previous similar message [ 1208.223275] Lustre: server umount lustre-MDT0000 complete [ 1208.938595] LustreError: 42398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1208.955537] LustreError: 42398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1210.857840] LustreError: 39938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814041 with bad export cookie 1683936042884126083 [ 1210.859520] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1210.873147] LustreError: 39938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1211.375817] LustreError: 39955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1211.376378] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1211.402672] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1214.061535] LustreError: 39954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1214.086795] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.27@tcp (stopping) [ 1219.182518] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.27@tcp (stopping) [ 1219.188834] Lustre: Skipped 1 previous similar message [ 1219.189834] LustreError: 40686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.213580] LustreError: 40686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1224.297638] LustreError: 39955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1224.322719] LustreError: 39955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1225.184140] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1225.443844] Lustre: server umount lustre-MDT0001 complete [ 1239.631311] Lustre: server umount lustre-OST0000 complete [ 1251.943667] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 1254.091961] Key type lgssc unregistered [ 1254.356942] LNet: 43826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1254.365308] LNetError: 43826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1254.384677] LNet: Removed LNI 192.168.206.127@tcp [ 1255.204534] Key type .llcrypt unregistered [ 1255.207168] Key type ._llcrypt unregistered [ 1272.606201] Key type ._llcrypt registered [ 1272.608674] Key type .llcrypt registered [ 1272.691609] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1273.553851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1273.634154] alg: No test for adler32 (adler32-zlib) [ 1274.594434] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 1274.742268] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 1276.400601] Key type lgssc registered [ 1277.207578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1282.601662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1282.615259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1283.956568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.561379] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1291.735981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.942424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1294.421777] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1299.498624] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1302.819498] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1307.024393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1307.260333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1308.280621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1311.240851] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1316.299557] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1321.961302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1321.964262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.976157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1321.994330] Lustre: Skipped 1 previous similar message [ 1324.158443] Lustre: lustre-OST0000: Not available for connect from 192.168.206.27@tcp (stopping) [ 1324.163110] Lustre: Skipped 1 previous similar message [ 1327.073250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1329.262065] Lustre: lustre-OST0000: Not available for connect from 192.168.206.27@tcp (stopping) [ 1329.266842] Lustre: Skipped 1 previous similar message [ 1334.376413] Lustre: lustre-OST0000: Not available for connect from 192.168.206.27@tcp (stopping) [ 1334.383139] Lustre: Skipped 2 previous similar messages [ 1336.288154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1336.401547] Lustre: server umount lustre-OST0000 complete [ 1352.676539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.685815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1352.689633] Lustre: Skipped 1 previous similar message [ 1358.877962] Lustre: server umount lustre-MDT0000 complete [ 1361.940507] LustreError: 45210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814192 with bad export cookie 12820073845613591375 [ 1361.941271] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1361.955820] LustreError: 45210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1368.240438] Lustre: server umount lustre-MDT0001 complete [ 1375.841228] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 1377.820647] Key type lgssc unregistered [ 1378.036195] LNet: 48940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.041423] LNetError: 48940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1378.056722] LNet: Removed LNI 192.168.206.127@tcp [ 1378.658735] Key type .llcrypt unregistered [ 1378.663771] Key type ._llcrypt unregistered [ 1394.474817] Key type ._llcrypt registered [ 1394.476351] Key type .llcrypt registered [ 1394.536269] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1395.131341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1395.208995] alg: No test for adler32 (adler32-zlib) [ 1396.195418] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 1396.396432] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 1398.024174] Key type lgssc registered [ 1398.766958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1403.712709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1403.725432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.972957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1408.189785] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1414.877775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.139640] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1415.287457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1420.066560] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1427.637488] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1432.233431] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1437.385166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1437.596956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1443.850222] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1450.939729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1453.881284] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1459.973791] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1464.069608] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1476.407880] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1513h18m24s [ 1477.594116] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1481.368655] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.27@tcp (stopping) [ 1481.701708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1481.702377] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.718124] Lustre: Skipped 2 previous similar messages [ 1482.421198] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.27@tcp (stopping) [ 1482.426825] Lustre: Skipped 3 previous similar messages [ 1485.877563] Lustre: server umount lustre-MDT0000 complete [ 1487.465575] LustreError: 51576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1487.478989] LustreError: 51576:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1488.700247] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1488.871156] LustreError: 50336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.904684] LustreError: 50336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1493.622406] LustreError: 50333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1494.590296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1494.764631] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1495.032954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1495.107568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1496.068294] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1499.115791] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1502.941751] Lustre: DEBUG MARKER: Wait for df (36743) ... [ 1504.392699] Lustre: DEBUG MARKER: done [ 1510.373372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1510.385358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.407551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1510.419379] Lustre: Skipped 3 previous similar messages [ 1511.401942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.948056] Lustre: server umount lustre-OST0000 complete [ 1521.634253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1521.645739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1521.659681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1521.669292] Lustre: Skipped 1 previous similar message [ 1521.700161] Lustre: Skipped 3 previous similar messages [ 1525.726671] Lustre: server umount lustre-MDT0000 complete [ 1529.324868] LustreError: 50337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1529.344839] LustreError: 50337:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1530.422475] LustreError: 52961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814361 with bad export cookie 8088685480707168754 [ 1530.426555] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1530.435690] LustreError: 52961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1531.086595] Lustre: server umount lustre-MDT0001 complete [ 1545.668597] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 1549.227623] Key type lgssc unregistered [ 1549.619148] LNet: 55260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.636132] LNetError: 55260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.693704] LNet: Removed LNI 192.168.206.127@tcp [ 1551.616963] Key type .llcrypt unregistered [ 1551.619636] Key type ._llcrypt unregistered [ 1591.272237] Key type ._llcrypt registered [ 1591.273975] Key type .llcrypt registered [ 1591.355560] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1592.554690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1592.581865] alg: No test for adler32 (adler32-zlib) [ 1593.600289] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 1593.907645] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 1595.648339] Key type lgssc registered [ 1597.238220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1606.878457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1606.899646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1608.194431] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1608.219542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1608.357587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1612.577533] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1620.785669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1620.936437] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1620.972719] Lustre: 57473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1621.185140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1625.064750] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1632.635425] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1638.918862] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1649.087184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1649.419131] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1649.947776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1658.522818] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1659.209534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1669.279731] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1676.763924] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1513h21m45s [ 1678.456332] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1686.111426] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1689.304721] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1689.350187] LustreError: 57481:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1709.720233] Lustre: lustre-MDT0000: Client 8387b6f0-b20d-462a-98e4-b460ad9960f5 (at 192.168.206.27@tcp) reconnecting [ 1729.416827] LustreError: 57481:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1732.079267] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1734.521235] Lustre: DEBUG MARKER: done [ 1741.293271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1741.303564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.317670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1742.244517] Lustre: server umount lustre-OST0000 complete [ 1746.411461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.425207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1746.445613] Lustre: Skipped 2 previous similar messages [ 1746.479416] Lustre: Skipped 2 previous similar messages [ 1747.560202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1747.570882] Lustre: Skipped 1 previous similar message [ 1751.981768] Lustre: server umount lustre-MDT0000 complete [ 1756.257425] LustreError: 58215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814587 with bad export cookie 7011479243856015481 [ 1756.282790] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1756.635092] Lustre: server umount lustre-MDT0001 complete [ 1768.273045] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 1770.673233] Key type lgssc unregistered [ 1770.901850] LNet: 61809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.905929] LNetError: 61809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.943189] LNet: Removed LNI 192.168.206.127@tcp [ 1772.805905] Key type .llcrypt unregistered [ 1772.809724] Key type ._llcrypt unregistered [ 1824.335598] Key type ._llcrypt registered [ 1824.338252] Key type .llcrypt registered [ 1824.486138] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1825.444855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1825.460900] alg: No test for adler32 (adler32-zlib) [ 1826.567711] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 1826.773475] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 1828.432133] Key type lgssc registered [ 1829.917570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1834.050403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1834.068469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1835.271449] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1835.311973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1835.417800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1836.260404] Lustre: Failing over lustre-MDT0000 [ 1836.442684] Lustre: server umount lustre-MDT0000 complete [ 1842.222312] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:24:31 (1786814671) [ 1848.703709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1849.029858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.426946] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1860.816583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.889471] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1860.926930] Lustre: 64607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1861.168036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1865.005231] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1871.575255] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1876.513813] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1881.929680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1882.140983] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1882.313955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.272303] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1890.392424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1894.374880] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1900.358290] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1907.170226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1907.178601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.199270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1910.754800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1910.756642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1910.772651] Lustre: Skipped 2 previous similar messages [ 1912.305668] Lustre: server umount lustre-MDT0000 complete [ 1915.664052] LustreError: 64600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814746 with bad export cookie 11331067951907480827 [ 1915.686551] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1915.874424] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1915.886620] Lustre: Skipped 1 previous similar message [ 1915.994551] Lustre: server umount lustre-MDT0001 complete [ 1919.663857] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1924.530282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1932.706350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1940.829530] LustreError: 66455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1940.844133] LustreError: 66455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1940.937489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.019948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1945.222911] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1946.085508] LustreError: 66456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1951.208714] LustreError: 66455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.946416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1954.396950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1955.439983] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1958.878049] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1965.735757] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1965.980324] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1971.878137] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1979.386034] Lustre: server umount lustre-MDT0000 complete [ 1979.876271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1979.887068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.898719] LustreError: 66456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1982.650118] LustreError: 66462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814813 with bad export cookie 11331067951907483102 [ 1982.650684] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1982.660702] LustreError: 66462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1982.887266] Lustre: server umount lustre-MDT0001 complete [ 1988.450753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1997.226531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.912261] Lustre: 63533:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814817/real 1786814817] req@00000000eae931e5 x1873610965843584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2002.933119] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.944497] Lustre: Skipped 2 previous similar messages [ 2003.936150] Lustre: 63535:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814818/real 1786814818] req@0000000091403665 x1873610965843712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814834 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2008.097241] Lustre: 63535:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814822/real 1786814822] req@0000000027e3cad0 x1873610965843840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814838 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2008.353084] LustreError: 66456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.373200] LustreError: 66456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2008.433327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.484252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2012.330205] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2013.536178] Lustre: 63533:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814828/real 1786814828] req@0000000057008ef5 x1873610965844224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814844 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2013.558329] Lustre: 63533:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2017.763300] LustreError: 71058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.789702] LustreError: 71058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2019.457117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.878545] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2024.589968] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2031.673889] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2037.171256] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2045.419031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2045.428071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2045.450462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2047.464364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2047.482199] Lustre: Skipped 2 previous similar messages [ 2050.210478] Lustre: server umount lustre-MDT0000 complete [ 2052.580116] LustreError: 71057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.600491] LustreError: 71057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2053.511769] LustreError: 64601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814884 with bad export cookie 11331067951907484481 [ 2053.521080] LustreError: 64601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2053.521570] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2053.736967] Lustre: server umount lustre-MDT0001 complete [ 2058.752929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2061.632769] Lustre: DEBUG MARKER: files compared the same [ 2072.369761] Lustre: server umount lustre-OST0000 complete [ 2083.874881] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 2086.051766] Key type lgssc unregistered [ 2086.313768] LNet: 74338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.330729] LNetError: 74338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2086.354763] LNet: Removed LNI 192.168.206.127@tcp [ 2086.929256] Key type .llcrypt unregistered [ 2086.931758] Key type ._llcrypt unregistered [ 2106.293657] Key type ._llcrypt registered [ 2106.295034] Key type .llcrypt registered [ 2106.382304] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 2106.987311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2107.115312] alg: No test for adler32 (adler32-zlib) [ 2108.201811] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 2108.415918] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 2110.124148] Key type lgssc registered [ 2111.266357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2120.028354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2120.046624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.555876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.597942] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug +malloc all [ 2131.942403] LustreError: 75734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.960568] LustreError: 75734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2135.411224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.684745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2139.677442] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug +malloc all [ 2146.656388] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2151.755336] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2157.980172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2158.223405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2164.205474] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug +malloc all [ 2166.449988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2171.753181] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2181.088889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2181.097299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2181.113332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2182.112776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2182.118677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2182.128895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.156319] Lustre: Skipped 1 previous similar message [ 2185.737733] Lustre: server umount lustre-OST0000 complete [ 2192.353160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2192.355482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.366800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2192.372172] Lustre: Skipped 1 previous similar message [ 2192.381027] Lustre: Skipped 1 previous similar message [ 2195.501597] Lustre: server umount lustre-MDT0000 complete [ 2197.474631] LustreError: 75735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.829922] LustreError: 75714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815029 with bad export cookie 8218882884227347160 [ 2198.833756] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2198.836546] LustreError: 75714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2199.298995] Lustre: server umount lustre-MDT0001 complete [ 2209.107953] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 2211.627956] Key type lgssc unregistered [ 2211.887897] LNet: 79268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2211.895192] LNetError: 79268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2211.915463] LNet: Removed LNI 192.168.206.127@tcp [ 2212.646937] Key type .llcrypt unregistered [ 2212.649519] Key type ._llcrypt unregistered [ 2243.190830] Key type ._llcrypt registered [ 2243.211488] Key type .llcrypt registered [ 2243.397962] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 2244.654713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2244.679515] alg: No test for adler32 (adler32-zlib) [ 2245.691518] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 2245.950580] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 2247.640201] Key type lgssc registered [ 2248.688210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2261.149589] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2261.186553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2276.832650] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2281.952338] LustreError: 80660:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.127@tcp: failed processing log, type 4: rc = -110 [ 2308.641448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2314.638982] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2323.535766] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2332.547416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.887275] Lustre: *** cfs_fail_loc=706, val=0*** [ 2333.208432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2333.291325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2338.281149] LustreError: 80686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.313559] LustreError: 80686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2338.482855] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2343.406419] LustreError: 80685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.440653] LustreError: 80685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2348.525461] LustreError: 80686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2348.553936] LustreError: 80686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2348.580426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.144907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2354.787815] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2362.131698] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2368.752435] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2374.117721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2374.122641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.132854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2374.627515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2374.635071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.661128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2377.430199] Lustre: server umount lustre-OST0000 complete [ 2384.871388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2384.872702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2384.874898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2384.885294] Lustre: Skipped 1 previous similar message [ 2388.208543] Lustre: server umount lustre-MDT0000 complete [ 2389.988709] LustreError: 81748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2390.011339] LustreError: 81748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2392.318281] LustreError: 80666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815223 with bad export cookie 6317989964994290489 [ 2392.321592] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2392.333127] LustreError: 80666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2392.535746] Lustre: server umount lustre-MDT0001 complete [ 2404.257835] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 2406.892202] Key type lgssc unregistered [ 2407.140863] LNet: 84281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2407.149319] LNetError: 84281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2408.181783] LNet: Removed LNI 192.168.206.127@tcp [ 2409.328279] Key type .llcrypt unregistered [ 2409.331966] Key type ._llcrypt unregistered [ 2435.417986] Key type ._llcrypt registered [ 2435.421114] Key type .llcrypt registered [ 2435.538741] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 2437.250812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2437.276049] alg: No test for adler32 (adler32-zlib) [ 2438.411145] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 2438.658270] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 2440.440543] Key type lgssc registered [ 2441.527939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2450.087568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2450.117352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2455.825490] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2464.700186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2465.318641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2469.404782] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2470.372813] LustreError: 86389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.391250] LustreError: 86389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2475.491549] LustreError: 86387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2475.513326] LustreError: 86387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2478.314524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2478.671421] LustreError: 86389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2478.718391] LustreError: 86389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2478.750702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2484.194742] LustreError: 86389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.210909] LustreError: 86389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2486.525889] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2489.325230] LustreError: 86393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.346863] LustreError: 86393:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2495.569315] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2495.579672] Lustre: MGS: Client eb271edd-05ac-4126-8cca-82fb7435c4db (at 0@lo) reconnecting [ 2495.608529] LustreError: 86403:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2495.926039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2496.000685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2501.233279] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2516.450181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.458751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2516.478462] Lustre: Skipped 1 previous similar message [ 2516.493458] Lustre: Skipped 1 previous similar message [ 2521.460429] Lustre: server umount lustre-OST0000 complete [ 2526.697826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2526.707102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2526.716035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2526.724369] Lustre: Skipped 1 previous similar message [ 2531.321589] Lustre: server umount lustre-MDT0000 complete [ 2531.812114] LustreError: 86392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.824766] LustreError: 86392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2535.525106] Lustre: server umount lustre-MDT0001 complete [ 2539.557740] Lustre: server umount MGS complete [ 2550.844575] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 13:36:20 (1786815380) [ 2579.753113] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 2589.353756] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 2627.415514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2632.152797] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2642.640038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.869780] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2642.913434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2643.021503] Lustre: lustre-MDT0001: new disk, initializing [ 2643.119353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2643.147797] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2643.161132] Lustre: 93572:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2643.174972] LustreError: 93572:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2648.217883] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2659.137130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2659.365183] Lustre: lustre-OST0000: new disk, initializing [ 2659.369376] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2659.376730] Lustre: 94339:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2659.446823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2665.273345] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2666.085918] Lustre: 94663:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000fe0180ed x1873611607547392/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786815512 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2674.583130] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2680.965499] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2680.973769] Lustre: MGS: Client 3322e082-bb88-456f-be91-e3dc8135ce7f (at 0@lo) reconnecting [ 2681.008957] Lustre: 92841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2681.024364] LustreError: 93558:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2681.060269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2681.130270] Lustre: lustre-MDT0000: new disk, initializing [ 2681.276410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2681.315063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2681.329146] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2682.402439] Lustre: 95398:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000002592e4f5 x1873611607554432/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786815529 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2682.413336] Lustre: 95398:0:(client.c:1639:after_reply()) Skipped 1 previous similar message [ 2685.655912] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2687.459474] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2691.501136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2691.509866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2691.522306] Lustre: Skipped 1 previous similar message [ 2691.535676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2717.153375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2717.164903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2717.181948] Lustre: Skipped 1 previous similar message [ 2717.193380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2718.177249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2719.201555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2719.210231] Lustre: Skipped 2 previous similar messages [ 2721.497486] Lustre: server umount lustre-OST0000 complete [ 2729.441109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2729.446847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.450190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2729.485592] Lustre: Skipped 2 previous similar messages [ 2739.681107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.692280] Lustre: Skipped 4 previous similar messages [ 2740.192167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2740.341324] Lustre: server umount lustre-MDT0000 complete [ 2750.445880] Lustre: server umount lustre-MDT0001 complete [ 2754.678077] Lustre: server umount MGS complete [ 2766.271533] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:39:55 (1786815595) [ 2782.958511] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 2785.631787] Key type lgssc unregistered [ 2785.989675] LNet: 97910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2786.007339] LNetError: 97910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2786.040512] LNet: Removed LNI 192.168.206.127@tcp [ 2787.112468] Key type .llcrypt unregistered [ 2787.115822] Key type ._llcrypt unregistered [ 2805.866886] Key type ._llcrypt registered [ 2805.869299] Key type .llcrypt registered [ 2805.975176] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 2806.837800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2806.862534] alg: No test for adler32 (adler32-zlib) [ 2807.906350] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 2808.156619] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 2809.832639] Key type lgssc registered [ 2811.121958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2819.166984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2819.179341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.706792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2825.823330] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2830.316187] LustreError: 99132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.351946] LustreError: 99132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2834.481260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2834.848761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2839.575867] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2847.451568] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2853.760881] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2861.482908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2861.997327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.101780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2868.782393] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2876.510312] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2890.727905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2890.730928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2890.754989] Lustre: Skipped 1 previous similar message [ 2890.779046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2893.021942] Lustre: server umount lustre-OST0000 complete [ 2900.965189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2900.969358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2900.981540] Lustre: Skipped 1 previous similar message [ 2900.996074] Lustre: Skipped 2 previous similar messages [ 2904.538997] Lustre: server umount lustre-MDT0000 complete [ 2906.088180] LustreError: 99133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2906.129174] LustreError: 99133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2908.682370] LustreError: 99113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815739 with bad export cookie 15028101065501842222 [ 2908.690350] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2908.697123] LustreError: 99113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2909.018458] Lustre: server umount lustre-MDT0001 complete [ 2920.407898] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 2923.412945] Key type lgssc unregistered [ 2923.652906] LNet: 102671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2923.665241] LNetError: 102671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2923.679135] LNet: Removed LNI 192.168.206.127@tcp [ 2924.701413] Key type .llcrypt unregistered [ 2924.703884] Key type ._llcrypt unregistered [ 2943.352324] Key type ._llcrypt registered [ 2943.354612] Key type .llcrypt registered [ 2943.472748] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 2944.673629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2944.804586] alg: No test for adler32 (adler32-zlib) [ 2945.865078] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 2946.064323] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 2947.744277] Key type lgssc registered [ 2949.047665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2954.530773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2954.548848] LustreError: 103754:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2954.551652] LustreError: 103755:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2954.567047] LustreError: 103754:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 2954.578135] LustreError: 103754:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2954.586955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.587536] LustreError: 103754:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2956.029951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2963.461939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.725122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2968.246061] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2973.512184] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing lsmod [ 2976.311515] LustreError: 104958:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2976.329476] LustreError: 104962:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2976.334657] LustreError: 104958:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 2976.341550] LustreError: 104958:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2976.351458] LustreError: 104958:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2976.352285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2976.710680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2981.859561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2981.863041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2981.882527] Lustre: Skipped 1 previous similar message [ 2981.893883] Lustre: Skipped 2 previous similar messages [ 2986.977545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2986.983801] Lustre: Skipped 2 previous similar messages [ 2989.027742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2989.045454] Lustre: Skipped 2 previous similar messages [ 2994.144128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2994.148440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2994.189355] Lustre: Skipped 2 previous similar messages [ 2994.520992] Lustre: server umount lustre-MDT0000 complete [ 2999.273878] LustreError: 104998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2999.295952] LustreError: 104998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2999.443089] LustreError: 103766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815830 with bad export cookie 1424020766825485769 [ 2999.449803] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2999.455519] LustreError: 103766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3001.378878] LustreError: 104224:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000eae931e5 x1873612139156992/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3001.578921] Lustre: server umount lustre-MDT0001 complete [ 3015.843949] Lustre: server umount lustre-OST0000 complete [ 3025.222724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3025.978307] LustreError: 106028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3026.044321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3031.522533] LustreError: 106029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3031.533726] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3031.537993] LustreError: 106029:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3035.624207] LustreError: 106028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.164197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.533655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3043.058637] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3049.873912] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3056.567583] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3064.148371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3064.250806] LustreError: 107850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3064.522114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3070.710209] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3070.968362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3077.602715] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3085.281302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3085.292150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3085.315620] Lustre: Skipped 1 previous similar message [ 3085.325907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3085.768074] Lustre: server umount lustre-OST0000 complete [ 3090.913806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3090.921569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3096.545220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3096.550686] Lustre: Skipped 4 previous similar messages [ 3099.712780] Lustre: server umount lustre-MDT0000 complete [ 3101.671062] LustreError: 106028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3101.692424] LustreError: 106028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3103.001973] LustreError: 107872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815933 with bad export cookie 1424020766825486455 [ 3103.008772] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3103.020751] LustreError: 107872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3103.631251] Lustre: server umount lustre-MDT0001 complete [ 3113.129591] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 3115.720419] Key type lgssc unregistered [ 3116.030295] LNet: 109565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3116.044032] LNetError: 109565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3116.056903] LNet: Removed LNI 192.168.206.127@tcp [ 3116.762706] Key type .llcrypt unregistered [ 3116.765118] Key type ._llcrypt unregistered [ 3138.459051] Key type ._llcrypt registered [ 3138.464944] Key type .llcrypt registered [ 3138.555835] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3139.587541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3139.844596] alg: No test for adler32 (adler32-zlib) [ 3140.869468] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 3141.085523] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 3142.754875] Key type lgssc registered [ 3143.681497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3152.585719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3152.602448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3154.080698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3159.011481] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3163.620429] LustreError: 110983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3163.634473] LustreError: 110983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3167.342853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3167.687721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3171.501340] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3178.363934] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3183.511988] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3189.303619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3189.684682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.163203] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3198.891193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3201.756868] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3206.373659] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3209.909795] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3214.171336] Lustre: server umount lustre-OST0000 complete [ 3218.913715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3218.919450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3218.928960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3219.436599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3219.438824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3224.547276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3224.559427] Lustre: Skipped 2 previous similar messages [ 3227.697448] Lustre: server umount lustre-MDT0000 complete [ 3229.665376] LustreError: 113526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.682430] LustreError: 113526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3230.691783] Lustre: 110294:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786816045/real 1786816045] req@0000000070dd4bda x1873612343635968/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786816061 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3230.720711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.733936] LustreError: 110977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.767922] LustreError: 110977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3231.066704] LustreError: 110963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816061 with bad export cookie 8194275370612417603 [ 3231.072909] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3231.082424] LustreError: 110963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3231.376668] Lustre: server umount lustre-MDT0001 complete [ 3242.378824] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 3245.385421] Key type lgssc unregistered [ 3245.615373] LNet: 114622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3245.626903] LNetError: 114622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3245.647935] LNet: Removed LNI 192.168.206.127@tcp [ 3246.451752] Key type .llcrypt unregistered [ 3246.456573] Key type ._llcrypt unregistered [ 3263.441446] Key type ._llcrypt registered [ 3263.448069] Key type .llcrypt registered [ 3263.589986] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3264.910019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3265.011898] alg: No test for adler32 (adler32-zlib) [ 3266.086048] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 3266.270995] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 3267.896171] Key type lgssc registered [ 3268.818044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3277.378048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3277.397864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3278.721581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3283.194203] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3289.070308] LustreError: 115844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3289.100293] LustreError: 115844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3291.195252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3291.532094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3295.141656] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3302.595151] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3308.519529] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3315.057895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3315.408827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3321.279970] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3323.633782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3328.349087] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3337.185916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3337.203235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3337.241705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3339.239668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3339.240703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3339.241707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3343.300016] Lustre: server umount lustre-OST0000 complete [ 3349.477440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.487945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3349.506623] Lustre: Skipped 1 previous similar message [ 3349.531895] Lustre: Skipped 2 previous similar messages [ 3354.594641] LustreError: 117092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.608407] LustreError: 117092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3354.651845] Lustre: server umount lustre-MDT0000 complete [ 3358.272748] LustreError: 115826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816189 with bad export cookie 12469119642717237646 [ 3358.281742] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3358.282067] LustreError: 115826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3358.571738] Lustre: server umount lustre-MDT0001 complete [ 3368.893730] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 3371.879524] Key type lgssc unregistered [ 3372.285684] LNet: 119381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3372.295495] LNetError: 119381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3372.330961] LNet: Removed LNI 192.168.206.127@tcp [ 3373.212165] Key type .llcrypt unregistered [ 3373.213785] Key type ._llcrypt unregistered [ 3395.819955] Key type ._llcrypt registered [ 3395.821965] Key type .llcrypt registered [ 3395.911873] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3397.335429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3397.354134] alg: No test for adler32 (adler32-zlib) [ 3398.510088] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 3398.753991] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 3400.416937] Key type lgssc registered [ 3401.514698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3409.417197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3409.432382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.867793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3415.360734] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3421.157107] LustreError: 120794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3421.192944] LustreError: 120794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3423.425302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.750766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3428.387704] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3435.412925] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3440.920843] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3446.681982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3446.960543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3451.935385] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3456.182283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3459.001921] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3465.543950] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3468.666692] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3468.674361] Lustre: Skipped 2 previous similar messages [ 3473.149227] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3473.155741] Lustre: Skipped 2 previous similar messages [ 3476.060471] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3476.089661] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3481.872284] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.27@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3481.885286] Lustre: Skipped 2 previous similar messages [ 3482.961178] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.27@tcp 192.168.0.[2,10]@tcp [ 3490.707249] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.27/24@tcp in log lustre-MDT0000 [ 3490.717815] Lustre: Skipped 2 previous similar messages [ 3497.060268] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.27/24@tcp [ 3497.071856] Lustre: Skipped 1 previous similar message [ 3502.564726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3502.574280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3502.581472] Lustre: Skipped 1 previous similar message [ 3507.681952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3507.696358] Lustre: Skipped 2 previous similar messages [ 3507.994760] Lustre: server umount lustre-OST0000 complete [ 3512.803585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3512.829898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3512.848285] Lustre: Skipped 1 previous similar message [ 3517.554023] Lustre: server umount lustre-MDT0000 complete [ 3517.927781] LustreError: 121795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3517.957719] LustreError: 121795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3520.973588] LustreError: 120774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816351 with bad export cookie 7825774591641366748 [ 3520.978031] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3520.982758] LustreError: 120774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3521.273888] Lustre: server umount lustre-MDT0001 complete [ 3531.205858] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 3533.587808] Key type lgssc unregistered [ 3533.833642] LNet: 125632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3533.846368] LNetError: 125632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3533.860369] LNet: Removed LNI 192.168.206.127@tcp [ 3534.709845] Key type .llcrypt unregistered [ 3534.713814] Key type ._llcrypt unregistered [ 3560.216886] Key type ._llcrypt registered [ 3560.221066] Key type .llcrypt registered [ 3560.341461] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3561.379503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3561.495087] alg: No test for adler32 (adler32-zlib) [ 3562.612428] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 3562.890924] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 3564.608441] Key type lgssc registered [ 3565.600682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3572.898214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3572.911835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.270394] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3574.288382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3574.297340] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.27/24@tcp [ 3578.670352] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3584.485305] LustreError: 127192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3584.514229] LustreError: 127192:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3586.499625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.797658] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3586.824394] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.27/24@tcp [ 3586.891926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3591.113310] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3598.481587] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3604.280492] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3611.284934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3611.548793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3616.822889] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3619.774954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3624.027787] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3635.169306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3635.174354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3635.179069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3635.184863] Lustre: Skipped 1 previous similar message [ 3640.303960] Lustre: server umount lustre-OST0000 complete [ 3645.412099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.413393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3645.424888] Lustre: Skipped 1 previous similar message [ 3645.436835] Lustre: Skipped 2 previous similar messages [ 3650.485665] Lustre: server umount lustre-MDT0000 complete [ 3650.529324] LustreError: 127192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3650.556910] LustreError: 127192:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3653.842114] LustreError: 129043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816484 with bad export cookie 14286810785455187281 [ 3653.850539] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3653.853160] LustreError: 129043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3654.120839] Lustre: server umount lustre-MDT0001 complete [ 3665.337780] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 3668.160640] Key type lgssc unregistered [ 3668.479967] LNet: 130781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3668.493097] LNetError: 130781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3668.516608] LNet: Removed LNI 192.168.206.127@tcp [ 3669.258867] Key type .llcrypt unregistered [ 3669.260928] Key type ._llcrypt unregistered [ 3693.366913] Key type ._llcrypt registered [ 3693.369404] Key type .llcrypt registered [ 3693.461686] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 3708.435664] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3709.563178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3709.596283] alg: No test for adler32 (adler32-zlib) [ 3710.604052] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 3710.829279] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 3712.504193] Key type lgssc registered [ 3713.703802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3744.543840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3744.567810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3745.805261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3745.832362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3745.912409] Lustre: lustre-MDT0000: new disk, initializing [ 3745.991473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3746.016134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3750.122696] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3762.022960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3762.192279] Lustre: 134184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3762.247892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3762.253353] Lustre: Skipped 1 previous similar message [ 3762.344913] Lustre: lustre-MDT0001: new disk, initializing [ 3762.446400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3762.506767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3762.526331] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3766.228996] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3775.139287] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3781.566308] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3788.882610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3789.267128] Lustre: lustre-OST0000: new disk, initializing [ 3789.270435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3789.275500] Lustre: 136153:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3789.381891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.540241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3796.554574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3796.675098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3796.810365] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3805.555459] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3811.175801] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3811.410578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3816.725621] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3816.937190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3822.561868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3822.564204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.570336] LustreError: Skipped 1 previous similar message [ 3822.584813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3822.592282] Lustre: Skipped 1 previous similar message [ 3822.617629] Lustre: Skipped 1 previous similar message [ 3827.685151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3827.692476] Lustre: Skipped 1 previous similar message [ 3832.824394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3832.840114] Lustre: Skipped 1 previous similar message [ 3833.317044] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3833.479867] Lustre: server umount lustre-OST0000 complete [ 3837.932032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3837.954303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.052953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.055055] Lustre: Skipped 1 previous similar message [ 3843.441479] Lustre: server umount lustre-MDT0000 complete [ 3846.845816] LustreError: 134177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816677 with bad export cookie 14650709221243406846 [ 3846.851352] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3846.859417] LustreError: 134177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3847.246582] Lustre: server umount lustre-MDT0001 complete [ 3855.532516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3856.002926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3860.275774] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3861.489208] LustreError: 138371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3861.511871] LustreError: 138371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3865.571905] LustreError: 138370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3867.555628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3867.850102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3871.578624] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3878.061789] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3882.661601] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3887.854299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3888.052723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3894.016391] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3900.934420] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3905.169370] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3905.422511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3909.707498] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3909.868412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3917.556915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3917.720299] Lustre: lustre-OST0001: new disk, initializing [ 3917.725301] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3917.738890] Lustre: 141739:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3917.861410] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3924.060393] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3924.111030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3924.115513] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3924.252509] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3932.708261] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3932.897212] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3937.552735] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3937.854448] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3944.178521] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8eb6533ee000.ost_server_uuid 50 [ 3945.739561] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8eb6533ee000.ost_server_uuid in IDLE state after 0 sec [ 3955.174081] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3955.181899] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3955.198115] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3955.200223] Lustre: Skipped 2 previous similar messages [ 3955.208159] Lustre: Skipped 2 previous similar messages [ 3956.876442] Lustre: server umount lustre-OST0001 complete [ 3960.294540] LustreError: 141747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3960.298113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3960.328076] LustreError: 141747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3960.343625] Lustre: Skipped 1 previous similar message [ 3965.426056] LustreError: 140189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3965.453020] LustreError: 140189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3966.582689] Lustre: server umount lustre-OST0000 complete [ 3970.531632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3975.656313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3975.663585] Lustre: Skipped 7 previous similar messages [ 3976.474864] Lustre: server umount lustre-MDT0000 complete [ 3980.005970] LustreError: 138352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816810 with bad export cookie 14650709221243407728 [ 3980.032342] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3980.032882] LustreError: 138352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3980.456159] Lustre: server umount lustre-MDT0001 complete [ 3997.332741] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 3999.916330] Key type lgssc unregistered [ 4000.178085] LNet: 144743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4000.184269] LNetError: 144743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4000.210412] LNet: Removed LNI 192.168.206.127@tcp [ 4001.069669] Key type .llcrypt unregistered [ 4001.073186] Key type ._llcrypt unregistered [ 4035.574493] Key type ._llcrypt registered [ 4035.576719] Key type .llcrypt registered [ 4035.655702] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 4050.557293] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 4051.451286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4051.670799] alg: No test for adler32 (adler32-zlib) [ 4052.724640] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 4052.973598] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 4054.624149] Key type lgssc registered [ 4055.512613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4086.251273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4086.283193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4087.448216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4087.498740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4087.558858] Lustre: lustre-MDT0000: new disk, initializing [ 4087.616267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4087.645735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4091.335579] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4099.744822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4099.847786] Lustre: 148910:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4099.899365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4099.903345] Lustre: Skipped 1 previous similar message [ 4099.962604] Lustre: lustre-MDT0001: new disk, initializing [ 4100.026082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4100.057936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4100.068571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4103.206242] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4111.546708] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4117.013530] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4123.062257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4123.294420] Lustre: lustre-OST0000: new disk, initializing [ 4123.300386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4123.306782] Lustre: 150879:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4123.385875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4128.339912] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4130.892649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4130.903817] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4131.028339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4137.487029] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4141.373559] Lustre: Failing over lustre-OST0000 [ 4141.544073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4141.546255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4141.549800] LustreError: Skipped 1 previous similar message [ 4141.565305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4141.575334] Lustre: Skipped 1 previous similar message [ 4143.502047] Lustre: server umount lustre-OST0000 complete [ 4161.539777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4162.016334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4162.047554] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4163.058963] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4167.960619] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4170.886083] Lustre: lustre-OST0000: Recovery over after 0:07, of 3 clients 3 recovered and 0 were evicted. [ 4170.887612] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4171.433796] Lustre: Failing over lustre-MDT0000 [ 4171.745444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4171.751878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4171.763433] Lustre: Skipped 1 previous similar message [ 4173.285671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4173.641511] Lustre: server umount lustre-MDT0000 complete [ 4178.406512] LustreError: 148917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4178.440579] LustreError: 148917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4181.095974] LustreError: 149298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4183.537000] LustreError: 149908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4183.554104] LustreError: 149908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4186.224904] LustreError: 148916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4189.730943] Lustre: 147136:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786817004/real 1786817004] req@00000000f20ac66c x1873613299952256/t0(0) o400->MGC192.168.206.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786817020 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4189.759388] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4190.749876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4192.362512] LustreError: 148918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4192.380850] LustreError: 148918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4199.906456] LustreError: 147132:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000069367593 x1873613299957632/t0(0) o250->MGC192.168.206.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4200.186837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4200.238192] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4202.601594] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4204.036321] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4205.558840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4205.571226] Lustre: Skipped 1 previous similar message [ 4205.594440] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4205.644052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4211.169261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4211.176050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4211.196787] Lustre: Skipped 1 previous similar message [ 4211.207328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4215.784184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4215.788191] Lustre: Skipped 1 previous similar message [ 4220.900919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4220.909672] Lustre: Skipped 2 previous similar messages [ 4224.992119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4225.129080] Lustre: server umount lustre-OST0000 complete [ 4231.138425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4231.138464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4231.149719] Lustre: Skipped 1 previous similar message [ 4231.161537] LustreError: Skipped 1 previous similar message [ 4231.181776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4231.188260] Lustre: Skipped 1 previous similar message [ 4241.377853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4241.389793] Lustre: Skipped 3 previous similar messages [ 4242.400867] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4242.632959] Lustre: server umount lustre-MDT0000 complete [ 4246.215878] LustreError: 148903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817077 with bad export cookie 9793031845889646673 [ 4246.223847] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4246.225624] LustreError: 148903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4246.494744] Lustre: server umount lustre-MDT0001 complete [ 4256.325096] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 4258.423799] Key type lgssc unregistered [ 4258.645596] LNet: 154541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4258.665704] LNetError: 154541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4258.693882] LNet: Removed LNI 192.168.206.127@tcp [ 4259.404778] Key type .llcrypt unregistered [ 4259.406585] Key type ._llcrypt unregistered [ 4291.572173] Key type ._llcrypt registered [ 4291.574303] Key type .llcrypt registered [ 4291.662849] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 4292.716798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4292.729093] alg: No test for adler32 (adler32-zlib) [ 4293.735884] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 4294.025030] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 4295.672166] Key type lgssc registered [ 4296.463948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4302.842506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4302.859325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4304.172279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4307.897238] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4314.596667] LustreError: 156060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4314.612235] LustreError: 156060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4315.780843] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4319.292775] LustreError: 156090:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4319.307822] LustreError: 156090:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4319.319843] LustreError: 156090:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 4319.595785] Lustre: server umount lustre-MDT0000 complete [ 4332.711579] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 4341.114389] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 4370.504647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4370.695122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4370.719529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4370.775449] Lustre: lustre-MDT0000: new disk, initializing [ 4370.830511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4370.851665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4373.772216] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4382.449206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4382.562900] Lustre: 159394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4382.614785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4382.619038] Lustre: Skipped 1 previous similar message [ 4382.679920] Lustre: lustre-MDT0001: new disk, initializing [ 4382.757580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4382.807728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4382.821123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4386.449605] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4393.979577] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4398.667573] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4403.890696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4404.121196] Lustre: lustre-OST0000: new disk, initializing [ 4404.127615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4404.135285] Lustre: 161359:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4404.214022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4409.280389] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4411.973320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4411.986960] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4412.092338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4416.668468] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4419.785743] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4421.271570] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:07:30 (1786817250) [ 4663.266478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4663.266649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4663.292140] Lustre: Skipped 1 previous similar message [ 4665.463386] Lustre: server umount lustre-MDT0000 complete [ 4668.384992] LustreError: 160553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4668.394614] LustreError: 160553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4668.424754] LustreError: 159385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817499 with bad export cookie 1283638348499909426 [ 4668.429152] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4668.432581] LustreError: 159385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4673.505038] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4673.506587] LustreError: 159400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4673.512968] Lustre: Skipped 1 previous similar message [ 4673.513514] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4673.534040] Lustre: Skipped 2 previous similar messages [ 4674.755131] Lustre: server umount lustre-MDT0001 complete [ 4684.353891] Lustre: server umount lustre-OST0000 complete [ 4694.388188] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 4705.101544] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 4740.759864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4741.019513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4741.053378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4741.127343] Lustre: lustre-MDT0000: new disk, initializing [ 4741.201674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4741.220523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4745.357536] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4755.252582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.410409] Lustre: 166016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4755.471790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4755.475905] Lustre: Skipped 1 previous similar message [ 4755.578556] Lustre: lustre-MDT0001: new disk, initializing [ 4755.687569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4755.716043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4755.725721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4759.440213] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4769.195627] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4775.890679] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4783.326174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4783.604602] Lustre: lustre-OST0000: new disk, initializing [ 4783.607763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4783.612727] Lustre: 167984:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4783.703609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4789.568644] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4790.858646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4790.880155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4790.981620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4800.760773] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4806.029980] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4806.475630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4810.257841] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4810.393367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4816.873097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4816.878759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4816.879678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4816.883756] Lustre: Skipped 1 previous similar message [ 4816.899089] Lustre: Skipped 1 previous similar message [ 4818.286918] Lustre: server umount lustre-OST0000 complete [ 4821.985373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4821.998198] Lustre: Skipped 1 previous similar message [ 4821.998380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4822.018147] Lustre: Skipped 1 previous similar message [ 4827.111054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4827.124230] Lustre: Skipped 1 previous similar message [ 4836.320172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4836.322494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4836.341990] Lustre: Skipped 3 previous similar messages [ 4836.556227] Lustre: server umount lustre-MDT0000 complete [ 4840.351660] LustreError: 166009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817671 with bad export cookie 1283638348500068249 [ 4840.361261] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4840.361962] LustreError: 166009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4840.696661] Lustre: server umount lustre-MDT0001 complete [ 4848.998976] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:14:38 (1786817678) [ 4864.811886] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 4874.966855] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 4913.404814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4913.705477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4913.746171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4913.801174] Lustre: lustre-MDT0000: new disk, initializing [ 4913.952302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4913.981771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4919.323750] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4929.564226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4929.733409] Lustre: 172742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4929.911857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4934.493387] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4943.691776] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4948.743106] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4954.939448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4955.235548] Lustre: lustre-OST0000: new disk, initializing [ 4955.238910] Lustre: Skipped 1 previous similar message [ 4955.244680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4955.250384] Lustre: Skipped 2 previous similar messages [ 4955.257521] Lustre: 174719:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4961.852876] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4963.912431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4963.923085] Lustre: Skipped 1 previous similar message [ 4963.931213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4964.013959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4971.780911] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4979.683207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4979.687023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4979.692071] LustreError: Skipped 2 previous similar messages [ 4979.693718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4979.728937] Lustre: Skipped 1 previous similar message [ 4985.210859] Lustre: server umount lustre-OST0000 complete [ 4989.927599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4994.803633] Lustre: server umount lustre-MDT0000 complete [ 4995.041414] LustreError: 172748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4995.054597] LustreError: 172748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4997.923261] LustreError: 172732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817828 with bad export cookie 1283638348500069138 [ 4997.928253] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4997.941642] LustreError: 172732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4998.198347] Lustre: server umount lustre-MDT0001 complete [ 5005.757839] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:17:15 (1786817835) [ 5020.900430] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 5029.870861] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 5065.137498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5065.570133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5065.611483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5065.692899] Lustre: lustre-MDT0000: new disk, initializing [ 5065.811615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5065.830469] Lustre: Skipped 2 previous similar messages [ 5065.889470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5071.284301] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5081.757328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5081.940951] Lustre: 179043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5082.321065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5086.477897] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5096.417868] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5101.617224] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5108.841558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5109.130311] Lustre: 181018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5114.547698] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5117.539190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5117.670465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5123.194622] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5133.285459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5133.286592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5133.306157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5133.306166] Lustre: Skipped 5 previous similar messages [ 5133.309598] Lustre: Skipped 2 previous similar messages [ 5135.861783] Lustre: server umount lustre-OST0000 complete [ 5143.524684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5143.541031] LustreError: Skipped 1 previous similar message [ 5145.572321] Lustre: server umount lustre-MDT0000 complete [ 5148.645574] LustreError: 181735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5148.664617] LustreError: 181735:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5148.997794] LustreError: 179033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817979 with bad export cookie 1283638348500070223 [ 5149.002044] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5149.004498] LustreError: 179033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5149.198970] Lustre: server umount lustre-MDT0001 complete [ 5159.248915] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 5162.670852] Key type lgssc unregistered [ 5162.975184] LNet: 182874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5162.981935] LNetError: 182874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5163.023317] LNet: Removed LNI 192.168.206.127@tcp [ 5164.206731] Key type .llcrypt unregistered [ 5164.209204] Key type ._llcrypt unregistered [ 5190.028666] Key type ._llcrypt registered [ 5190.030606] Key type .llcrypt registered [ 5190.171916] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 5191.110993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5191.155074] alg: No test for adler32 (adler32-zlib) [ 5192.175579] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 5192.507915] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 5194.384248] Key type lgssc registered [ 5195.858750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5204.734993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5204.749240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5206.179328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5210.667984] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5216.737710] LustreError: 184293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5216.761059] LustreError: 184293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5218.294301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.680377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5223.244256] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5230.435319] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5235.698160] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5241.986866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5242.454616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5249.780359] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5251.721823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5257.751332] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5277.665742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5277.670639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5277.680112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5277.699851] Lustre: Skipped 1 previous similar message [ 5277.715982] Lustre: Skipped 1 previous similar message [ 5281.913449] Lustre: server umount lustre-OST0000 complete [ 5287.906259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.925790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5287.930671] Lustre: Skipped 1 previous similar message [ 5292.026616] Lustre: server umount lustre-MDT0000 complete [ 5293.031769] LustreError: 184294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5293.065861] LustreError: 184294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5295.773472] LustreError: 184273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818126 with bad export cookie 6088304272944097286 [ 5295.779064] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5295.786967] LustreError: 184273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5296.234609] Lustre: server umount lustre-MDT0001 complete [ 5305.718547] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 5308.419525] Key type lgssc unregistered [ 5308.696895] LNet: 187850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5308.701660] LNetError: 187850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5308.745117] LNet: Removed LNI 192.168.206.127@tcp [ 5309.492158] Key type .llcrypt unregistered [ 5309.493971] Key type ._llcrypt unregistered [ 5336.143987] Key type ._llcrypt registered [ 5336.145359] Key type .llcrypt registered [ 5336.287874] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 5338.019128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5338.078978] alg: No test for adler32 (adler32-zlib) [ 5339.105489] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 5339.274219] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 5340.952148] Key type lgssc registered [ 5342.134242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5348.943920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5348.959033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5350.362344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5354.561538] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5360.623230] LustreError: 189269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5360.635951] LustreError: 189269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5363.164948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5363.552143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5368.815808] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5377.940547] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5384.088355] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5390.811239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5391.247490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5395.352435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5396.637398] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5404.176133] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5415.911997] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5415.920984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5415.930853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5417.102762] Lustre: lustre-OST0000: Not available for connect from 192.168.206.27@tcp (stopping) [ 5418.979967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5418.984299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5418.988495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5419.014878] Lustre: Skipped 1 previous similar message [ 5422.185083] Lustre: lustre-OST0000: Not available for connect from 192.168.206.27@tcp (stopping) [ 5427.311828] Lustre: lustre-OST0000: Not available for connect from 192.168.206.27@tcp (stopping) [ 5427.316178] Lustre: Skipped 2 previous similar messages [ 5428.192137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5428.491957] Lustre: server umount lustre-OST0000 complete [ 5436.309944] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8eb644423000.ost_server_uuid 50 [ 5438.180396] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5454.829692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.842274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5454.852274] Lustre: Skipped 1 previous similar message [ 5457.503646] Lustre: server umount lustre-MDT0000 complete [ 5459.939664] LustreError: 189268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.962029] LustreError: 189268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5461.705960] LustreError: 192319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818292 with bad export cookie 6248753187231023541 [ 5461.708857] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5461.720046] LustreError: 192319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5462.209200] Lustre: server umount lustre-MDT0001 complete [ 5471.770623] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:25:01 (1786818301) [ 5479.464898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5479.833456] LustreError: 193191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5479.885395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5484.646180] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5485.026253] LustreError: 193192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5490.146690] LustreError: 193191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5492.561261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5492.826290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5496.926822] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5503.216898] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5508.339901] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5514.011490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5514.278969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5519.441073] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5525.559206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5525.957873] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5532.455313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5532.664397] Lustre: lustre-OST0001: new disk, initializing [ 5532.671750] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5532.684328] Lustre: 196032:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5532.819206] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5538.654720] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5540.405255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5540.416810] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5540.526499] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5546.379984] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5551.079551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5551.085058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5551.092484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5551.106356] Lustre: Skipped 1 previous similar message [ 5551.110725] Lustre: Skipped 1 previous similar message [ 5554.837611] Lustre: server umount lustre-OST0000 complete [ 5556.204317] LustreError: 195011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5556.223059] LustreError: 195011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5560.202968] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5560.467845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5564.311705] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5564.455479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5572.719187] LustreError: 195010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5572.738019] LustreError: 195010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5581.799482] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5581.800876] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5581.804473] LustreError: Skipped 1 previous similar message [ 5581.816541] Lustre: Skipped 1 previous similar message [ 5583.710121] Lustre: server umount lustre-OST0001 complete [ 5586.923555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5586.928492] Lustre: Skipped 5 previous similar messages [ 5601.249968] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5601.378991] Lustre: server umount lustre-MDT0000 complete [ 5604.871088] LustreError: 193171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818435 with bad export cookie 6248753187231024612 [ 5604.879944] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5604.884422] LustreError: 193171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5605.188807] Lustre: server umount lustre-MDT0001 complete [ 5625.885773] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:27:35 (1786818455) [ 5631.969221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5632.251673] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5632.300531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5632.471174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5636.235307] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5643.836566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5643.989072] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5644.052865] Lustre: 199470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5648.609186] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5655.214406] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5660.254959] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5665.800101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5665.994686] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5671.593485] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5675.097618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5680.400628] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5688.386976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5688.477657] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5695.502976] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5696.251583] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5702.944659] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5706.738730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5706.750787] Lustre: Skipped 3 previous similar messages [ 5706.753881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5706.760804] Lustre: Skipped 7 previous similar messages [ 5712.717069] Lustre: server umount lustre-OST0000 complete [ 5714.033513] LustreError: 201818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.046177] LustreError: 201818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5732.324837] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5732.340358] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5732.342206] LustreError: Skipped 1 previous similar message [ 5732.360657] Lustre: Skipped 1 previous similar message [ 5734.568330] Lustre: server umount lustre-OST0001 complete [ 5741.543234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5744.853990] Lustre: server umount lustre-MDT0000 complete [ 5748.508751] LustreError: 199462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818579 with bad export cookie 6248753187231026019 [ 5748.511486] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5748.516915] LustreError: 199462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5749.116802] Lustre: server umount lustre-MDT0001 complete [ 5772.381926] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:30:01 (1786818601) [ 5788.640257] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 5796.270860] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 5827.972700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5828.157769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5828.187298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5828.240530] Lustre: lustre-MDT0000: new disk, initializing [ 5828.318901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5828.326233] Lustre: Skipped 3 previous similar messages [ 5828.341112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5832.663443] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5842.586926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5842.717951] Lustre: 207568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5842.805567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5842.817152] Lustre: Skipped 1 previous similar message [ 5842.925473] Lustre: lustre-MDT0001: new disk, initializing [ 5843.116027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5843.138434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5847.772518] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5856.940825] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5862.554527] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5869.630550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5869.915369] Lustre: lustre-OST0000: new disk, initializing [ 5869.921540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5869.930621] Lustre: 209535:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5875.446766] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5876.800862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5876.808790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5876.948114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5884.263488] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5888.838882] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5889.016592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5893.398172] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5893.570929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5897.700756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5897.701126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5897.708553] LustreError: Skipped 1 previous similar message [ 5897.714725] Lustre: Skipped 2 previous similar messages [ 5897.724614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5897.730773] Lustre: Skipped 9 previous similar messages [ 5901.285238] Lustre: server umount lustre-OST0000 complete [ 5910.524450] Lustre: server umount lustre-MDT0000 complete [ 5913.058274] LustreError: 208466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5913.084683] LustreError: 208466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 5913.471164] LustreError: 207560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818744 with bad export cookie 6248753187231027804 [ 5913.477245] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5913.478938] LustreError: 207560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5913.717301] Lustre: server umount lustre-MDT0001 complete [ 5921.137943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5925.751716] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5933.396345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5938.396713] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5945.491450] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5951.690675] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5958.329705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5958.575392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5958.585690] Lustre: Skipped 4 previous similar messages [ 5964.286670] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5972.115139] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5978.328306] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5978.660152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5983.497280] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5983.828575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5987.306334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5987.307425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5987.326974] LustreError: Skipped 1 previous similar message [ 5987.364773] Lustre: Skipped 3 previous similar messages [ 5992.454289] Lustre: server umount lustre-OST0000 complete [ 5998.921402] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6002.974718] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6030.914643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6037.482430] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6044.668517] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6050.267494] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6052.727285] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.738746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.742963] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6052.765437] Lustre: Skipped 1 previous similar message [ 6057.980205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6069.216124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6069.396966] Lustre: server umount lustre-OST0000 complete [ 6079.617053] Lustre: server umount lustre-MDT0000 complete [ 6083.259982] LustreError: 211730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818914 with bad export cookie 6248753187231028686 [ 6083.268707] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6083.275076] LustreError: 211730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6083.483987] Lustre: server umount lustre-MDT0001 complete [ 6092.237281] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:35:21 (1786818921) [ 6099.501632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6099.882111] LustreError: 217717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.905463] LustreError: 217717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6104.745312] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6112.988663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6117.383131] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6124.432387] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6129.500534] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6135.568520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6141.652756] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6148.534651] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6153.052437] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6153.273194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6157.558957] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6157.811283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6163.923197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6164.122667] Lustre: lustre-OST0001: new disk, initializing [ 6164.127448] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6164.134850] Lustre: 221088:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6169.687659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6169.692629] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6169.700628] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6169.768440] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6179.361147] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6183.636462] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6183.855444] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6188.058567] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6188.368253] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6190.567615] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6190.580413] LustreError: Skipped 2 previous similar messages [ 6190.584059] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6190.590330] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6190.590337] Lustre: Skipped 14 previous similar messages [ 6190.617236] Lustre: Skipped 5 previous similar messages [ 6196.478762] Lustre: server umount lustre-OST0001 complete [ 6203.147762] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6203.376100] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6207.723274] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6207.957619] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6236.130195] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6236.371571] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6236.379491] Lustre: Skipped 5 previous similar messages [ 6241.770208] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6241.786205] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6241.797162] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6241.806568] Lustre: Skipped 1 previous similar message [ 6241.914314] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6249.448681] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6251.137706] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6267.371291] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6267.510779] Lustre: server umount lustre-OST0001 complete [ 6298.081520] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6298.287342] Lustre: server umount lustre-OST0000 complete [ 6308.075995] Lustre: server umount lustre-MDT0000 complete [ 6311.412121] LustreError: 217699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819142 with bad export cookie 6248753187231029652 [ 6311.421205] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6311.426425] LustreError: 217699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6317.822555] Lustre: server umount lustre-MDT0001 complete [ 6339.878463] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:39:29 (1786819169) [ 6345.972328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6346.323960] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6346.350463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6351.742252] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6360.179828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.330710] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6360.367156] Lustre: 226333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6365.047882] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6372.342962] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6377.160396] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6382.778587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6383.030530] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6387.778815] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6394.019204] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6402.202723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6402.308475] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6408.629115] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6415.311380] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6416.846218] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6420.656983] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6420.885772] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6425.088716] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6425.250167] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6429.845991] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8eb651a4b000.ost_server_uuid 50 [ 6431.245993] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8eb651a4b000.ost_server_uuid in IDLE state after 0 sec [ 6431.996907] Lustre: Permanently deactivating lustre-OST0001 [ 6432.006259] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6434.919544] Lustre: Permanently reactivating lustre-OST0001 [ 6444.120622] Lustre: server umount lustre-OST0001 complete [ 6444.523597] LustreError: 228181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6444.542139] LustreError: 228181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [ 6447.593801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6447.603675] LustreError: Skipped 7 previous similar messages [ 6447.610945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6447.625586] Lustre: Skipped 8 previous similar messages [ 6453.298368] Lustre: server umount lustre-OST0000 complete [ 6462.503817] Lustre: server umount lustre-MDT0000 complete [ 6465.632708] LustreError: 226327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819296 with bad export cookie 6248753187231030940 [ 6465.638026] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6465.643383] LustreError: 226327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6465.912237] Lustre: server umount lustre-MDT0001 complete [ 6487.557328] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:41:56 (1786819316) [ 6497.026994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6497.387629] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6497.408476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6497.420391] Lustre: Skipped 5 previous similar messages [ 6502.382320] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6510.294675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6510.405517] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6510.422899] Lustre: 233024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6514.038555] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6519.659155] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6524.746564] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6530.893204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6531.057119] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6531.075272] Lustre: Permanently deactivating lustre-OST0000 [ 6531.083821] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6536.311144] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6540.816294] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6541.962203] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6547.615949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6547.662664] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6548.777208] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6552.518608] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6558.455657] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6560.701340] Lustre: Permanently reactivating lustre-OST0000 [ 6560.738526] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6560.744148] Lustre: Skipped 2 previous similar messages [ 6567.981164] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6567.998874] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6568.017928] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6568.095224] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6568.138971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6587.872115] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6588.073237] Lustre: server umount lustre-OST0001 complete [ 6598.463540] Lustre: server umount lustre-OST0000 complete [ 6608.080886] Lustre: server umount lustre-MDT0000 complete [ 6611.607055] LustreError: 233016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819442 with bad export cookie 6248753187231032592 [ 6611.616620] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6611.623900] LustreError: 233016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6611.994763] Lustre: server umount lustre-MDT0001 complete [ 6622.492900] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 6625.600335] Key type lgssc unregistered [ 6625.973610] LNet: 237913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6625.979555] LNetError: 237913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6625.993326] LNet: Removed LNI 192.168.206.127@tcp [ 6626.665990] Key type .llcrypt unregistered [ 6626.670347] Key type ._llcrypt unregistered [ 6647.862693] Key type ._llcrypt registered [ 6647.872912] Key type .llcrypt registered [ 6648.081435] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 6649.590460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6649.614181] alg: No test for adler32 (adler32-zlib) [ 6650.613942] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 6650.811433] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 6652.440138] Key type lgssc registered [ 6653.433520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6662.859325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6662.891838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6664.235139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6668.457939] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6674.404360] LustreError: 239373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6674.412829] LustreError: 239373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6675.544399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6675.624132] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6675.650484] Lustre: 239360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6675.660690] Lustre: Permanently deactivating lustre-MDT0001 [ 6675.675417] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6676.003966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6679.921069] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6679.953152] LustreError: 239404:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6679.979536] LustreError: 239404:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6679.994532] LustreError: 239404:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 6681.546178] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6688.577922] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6694.814428] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6701.155523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6701.547510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6706.921441] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6709.741668] LustreError: 241216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.754132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6713.362206] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6714.853688] LustreError: 241217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6719.473069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6719.790780] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6720.832171] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6720.852248] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6726.856250] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6734.991051] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6738.722572] Lustre: Permanently reactivating lustre-MDT0001 [ 6738.740587] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6738.744815] Lustre: Skipped 2 previous similar messages [ 6742.128986] Lustre: lustre-MDT0001: Client 38b31e8d-f859-4d4c-a0a6-e8383c4516ff (at 192.168.206.27@tcp) reconnecting [ 6748.098286] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6749.646987] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6755.061450] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6755.272910] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6759.532386] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6759.751501] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6762.368964] Lustre: Permanently deactivating lustre-MDT0001 [ 6762.387776] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6762.393298] Lustre: Skipped 2 previous similar messages [ 6771.268646] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6778.117524] Lustre: server umount lustre-MDT0000 complete [ 6778.349110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6778.368568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6778.380024] LustreError: 239373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6780.387084] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6780.401901] Lustre: Skipped 1 previous similar message [ 6781.596292] LustreError: 239353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819612 with bad export cookie 386242299801701012 [ 6781.596758] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6781.608142] LustreError: 239353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6781.947406] Lustre: server umount lustre-MDT0001 complete [ 6785.744351] Lustre: server umount lustre-OST0000 complete [ 6796.896650] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:47:06 (1786819626) [ 6806.131664] Lustre: server umount lustre-OST0001 complete [ 6812.196430] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 6821.104714] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 6862.656195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6862.805263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6862.818634] Lustre: Skipped 2 previous similar messages [ 6862.835086] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6862.890710] Lustre: lustre-MDT0000: new disk, initializing [ 6862.945213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6862.960885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6867.630566] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6876.940701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.035588] Lustre: 247633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6877.075262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6877.079105] Lustre: Skipped 1 previous similar message [ 6877.134443] Lustre: lustre-MDT0001: new disk, initializing [ 6877.200717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6877.237798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6877.253367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6881.509505] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6889.792616] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6895.843287] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6901.466755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6901.762552] Lustre: lustre-OST0000: new disk, initializing [ 6901.767484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6901.778267] Lustre: 249605:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6901.888242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6907.411337] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6910.033253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6910.053119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6910.142937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6915.690984] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6917.861072] LustreError: 247638:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6926.071984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6926.286537] Lustre: lustre-OST0001: new disk, initializing [ 6926.291655] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6926.299196] Lustre: 250719:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6927.904393] LustreError: 247638:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6927.923970] LustreError: 250430:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6931.988580] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6932.045396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6932.057535] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6932.145639] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6938.000201] LustreError: 250430:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6941.506117] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6942.772773] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6945.919130] Lustre: lustre-OST0001: Not available for connect from 192.168.206.27@tcp (stopping) [ 6947.810498] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6947.829757] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6947.833124] LustreError: Skipped 1 previous similar message [ 6947.835837] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6947.868932] Lustre: Skipped 2 previous similar messages [ 6950.549513] Lustre: server umount lustre-OST0001 complete [ 6951.019372] LustreError: 249589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6951.037259] LustreError: 249589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6963.191738] LustreError: 250409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6963.206415] LustreError: 250409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6968.301531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6968.321384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6968.332361] Lustre: Skipped 2 previous similar messages [ 6973.239148] Lustre: server umount lustre-OST0000 complete [ 6978.531937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6978.544268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6978.548143] Lustre: Skipped 2 previous similar messages [ 6978.566120] Lustre: Skipped 1 previous similar message [ 6983.652070] LustreError: 247638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6983.669074] LustreError: 247638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6983.686686] Lustre: server umount lustre-MDT0000 complete [ 6987.887028] LustreError: 247624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819818 with bad export cookie 386242299801703609 [ 6987.893073] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6987.913096] LustreError: 247624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6994.479702] Lustre: server umount lustre-MDT0001 complete [ 7005.838057] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 7008.271027] Key type lgssc unregistered [ 7008.559182] LNet: 252760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7008.585312] LNetError: 252760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7009.638943] LNet: Removed LNI 192.168.206.127@tcp [ 7010.447249] Key type .llcrypt unregistered [ 7010.449352] Key type ._llcrypt unregistered [ 7050.796980] Key type ._llcrypt registered [ 7050.801057] Key type .llcrypt registered [ 7050.939371] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 7052.257245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7052.319308] alg: No test for adler32 (adler32-zlib) [ 7053.484716] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 7053.787946] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 7055.472185] Key type lgssc registered [ 7056.791140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7064.806995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7064.844948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7066.092307] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7066.114403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7066.242811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7070.227381] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7077.999210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7078.081453] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7078.106813] Lustre: 254972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7078.375660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7082.675666] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7091.885877] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7098.006540] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7105.436945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7105.544819] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7105.707934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7111.234963] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7111.800884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7118.264715] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7171.558961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7171.570788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7171.583852] Lustre: Skipped 1 previous similar message [ 7176.047910] Lustre: server umount lustre-OST0000 complete [ 7179.718729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7191.232394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7191.638798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7192.682624] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7192.700395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7192.701788] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7192.732411] Lustre: Skipped 1 previous similar message [ 7197.287898] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7204.244256] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7244.259776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7244.260848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7244.269803] LustreError: Skipped 1 previous similar message [ 7244.271802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7244.276326] Lustre: Skipped 2 previous similar messages [ 7249.381606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7249.395364] Lustre: Skipped 2 previous similar messages [ 7250.393638] Lustre: server umount lustre-OST0000 complete [ 7254.509392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7254.521764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7254.525074] Lustre: Skipped 1 previous similar message [ 7254.556701] Lustre: Skipped 1 previous similar message [ 7259.619218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7259.636686] Lustre: Skipped 1 previous similar message [ 7260.819992] Lustre: server umount lustre-MDT0000 complete [ 7264.542863] LustreError: 254963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820095 with bad export cookie 17921728990480658095 [ 7264.546348] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7264.548026] LustreError: 254963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7264.727767] Lustre: server umount lustre-MDT0001 complete [ 7274.693160] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 7277.345653] Key type lgssc unregistered [ 7277.599423] LNet: 260400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7277.604994] LNetError: 260400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7277.616812] LNet: Removed LNI 192.168.206.127@tcp [ 7278.303623] Key type .llcrypt unregistered [ 7278.308967] Key type ._llcrypt unregistered [ 7301.232172] Key type ._llcrypt registered [ 7301.234787] Key type .llcrypt registered [ 7301.316812] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 7302.986534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7303.065415] alg: No test for adler32 (adler32-zlib) [ 7304.173882] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 7304.412147] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 7306.120180] Key type lgssc registered [ 7307.546403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7315.779619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7315.822267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7317.137680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7321.538711] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7326.698892] LustreError: 261811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.729952] LustreError: 261811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7330.242040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7330.762579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7335.294641] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7343.040082] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7350.699723] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7357.272414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7357.623674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7361.711783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7363.290840] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7370.382798] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7396.835269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7396.837954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7396.848549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7401.958297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7401.960793] Lustre: Skipped 1 previous similar message [ 7403.220866] Lustre: server umount lustre-OST0000 complete [ 7407.586423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7407.594753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7407.610959] Lustre: Skipped 1 previous similar message [ 7407.615363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7407.620324] Lustre: Skipped 1 previous similar message [ 7412.193398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7412.198107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7414.121544] Lustre: server umount lustre-MDT0000 complete [ 7417.318676] LustreError: 262820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7417.351689] LustreError: 262820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7417.760931] LustreError: 261792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820248 with bad export cookie 11306770172364733369 [ 7417.764207] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7417.776730] LustreError: 261792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7418.356614] Lustre: server umount lustre-MDT0001 complete [ 7428.908186] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 7431.226823] Key type lgssc unregistered [ 7431.472181] LNet: 266119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7431.480449] LNetError: 266119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7431.502210] LNet: Removed LNI 192.168.206.127@tcp [ 7432.310430] Key type .llcrypt unregistered [ 7432.312584] Key type ._llcrypt unregistered [ 7446.521594] Key type ._llcrypt registered [ 7446.526403] Key type .llcrypt registered [ 7446.650945] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 7447.531731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7447.567119] alg: No test for adler32 (adler32-zlib) [ 7448.518709] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 7448.768476] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 7450.432240] Key type lgssc registered [ 7451.277322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7458.486522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7458.502312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7459.919423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7464.420850] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7470.051900] LustreError: 267322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7470.084065] LustreError: 267322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7473.565257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7474.105823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7478.338697] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7485.191063] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7490.146568] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7496.046521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7496.452268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7501.594648] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7504.627936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7509.067288] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7519.205173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7519.214142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7519.227499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7520.227742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.230382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7520.248741] Lustre: Skipped 1 previous similar message [ 7524.275461] Lustre: server umount lustre-OST0000 complete [ 7530.465963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7530.470863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7530.471579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7530.483958] Lustre: Skipped 1 previous similar message [ 7533.864779] Lustre: server umount lustre-MDT0000 complete [ 7535.585425] LustreError: 267321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7535.608341] LustreError: 267321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7537.161479] LustreError: 267302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820368 with bad export cookie 4397010722569846021 [ 7537.165429] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7537.170849] LustreError: 267302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7537.520713] Lustre: server umount lustre-MDT0001 complete [ 7547.224644] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 7549.488894] Key type lgssc unregistered [ 7549.755814] LNet: 271001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7549.767666] LNetError: 271001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7549.805408] LNet: Removed LNI 192.168.206.127@tcp [ 7550.587532] Key type .llcrypt unregistered [ 7550.590188] Key type ._llcrypt unregistered [ 7571.717437] Key type ._llcrypt registered [ 7571.719954] Key type .llcrypt registered [ 7571.819371] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 7572.833440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7572.848965] alg: No test for adler32 (adler32-zlib) [ 7573.868305] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 7574.059753] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 7575.728619] Key type lgssc registered [ 7576.790095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7583.518286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7583.530456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7584.853564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7587.757447] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7593.583665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7593.852690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7597.825803] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7605.348860] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7610.526925] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7616.563210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7616.839884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7622.582279] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7628.553267] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7630.209917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7654.891316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7654.898429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.911235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7655.911381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.912463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7655.917346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7658.209269] Lustre: server umount lustre-OST0000 complete [ 7665.634872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7665.643465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7665.662087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7665.675688] Lustre: Skipped 1 previous similar message [ 7668.710803] Lustre: server umount lustre-MDT0000 complete [ 7671.271629] LustreError: 275916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7671.284273] LustreError: 275916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7672.109738] LustreError: 272399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820502 with bad export cookie 15536050677216775465 [ 7672.113616] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7672.124135] LustreError: 272399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7672.377580] Lustre: server umount lustre-MDT0001 complete [ 7681.641577] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 7683.944595] Key type lgssc unregistered [ 7684.295651] LNet: 276738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7684.301663] LNetError: 276738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7684.320522] LNet: Removed LNI 192.168.206.127@tcp [ 7685.015631] Key type .llcrypt unregistered [ 7685.019288] Key type ._llcrypt unregistered [ 7698.835811] Key type ._llcrypt registered [ 7698.839141] Key type .llcrypt registered [ 7698.934622] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 7699.747336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7699.762793] alg: No test for adler32 (adler32-zlib) [ 7700.776601] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 7700.980960] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 7702.624983] Key type lgssc registered [ 7703.446250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7710.162769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7710.188547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7711.591605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7715.371681] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7721.959941] LustreError: 277956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7721.976362] LustreError: 277956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7722.197079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7722.462200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7725.896267] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7732.016526] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7736.911273] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7744.234493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7744.648141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7750.089440] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7756.993411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7757.044567] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7768.548888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7768.557742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7768.583120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7772.643305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7772.647907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7772.649736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7774.639753] Lustre: server umount lustre-OST0000 complete [ 7778.786977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7778.797036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7778.814160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7778.825334] Lustre: Skipped 1 previous similar message [ 7782.883687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7782.887021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7784.450209] Lustre: server umount lustre-MDT0000 complete [ 7787.682366] LustreError: 277921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820618 with bad export cookie 7588439951774145666 [ 7787.687985] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7787.692327] LustreError: 277921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7787.973184] Lustre: server umount lustre-MDT0001 complete [ 7798.399960] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 7801.108991] Key type lgssc unregistered [ 7801.451643] LNet: 281658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7801.461272] LNetError: 281658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7801.479463] LNet: Removed LNI 192.168.206.127@tcp [ 7802.267435] Key type .llcrypt unregistered [ 7802.272585] Key type ._llcrypt unregistered [ 7812.434421] Key type ._llcrypt registered [ 7812.437529] Key type .llcrypt registered [ 7812.568859] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7829.088486] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 7842.658211] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 7843.511462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7843.537177] alg: No test for adler32 (adler32-zlib) [ 7844.554858] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 7844.733977] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 7846.368929] Key type lgssc registered [ 7847.252603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7877.082201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7877.123538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7878.446863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7878.495767] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7878.583093] Lustre: lustre-MDT0000: new disk, initializing [ 7878.647547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7878.696505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7882.442404] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7892.132859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7892.227817] Lustre: 285203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7892.303893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7892.307436] Lustre: Skipped 1 previous similar message [ 7892.368606] Lustre: lustre-MDT0001: new disk, initializing [ 7892.453911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7892.508366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7892.531397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7896.943564] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7904.924855] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7909.733815] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7916.113667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7916.342682] Lustre: lustre-OST0000: new disk, initializing [ 7916.346488] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7916.352887] Lustre: 287167:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7916.435134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7921.795177] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7922.835281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7922.850700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7922.953156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7931.457914] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7935.422312] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7935.602722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7939.647794] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7939.862812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7943.659072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7943.662017] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7943.663230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7943.690691] Lustre: Skipped 1 previous similar message [ 7943.699527] Lustre: Skipped 1 previous similar message [ 7948.124936] Lustre: server umount lustre-OST0000 complete [ 7953.889278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7953.890845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.900040] LustreError: Skipped 1 previous similar message [ 7953.900574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7953.919195] Lustre: Skipped 1 previous similar message [ 7957.224689] Lustre: server umount lustre-MDT0000 complete [ 7959.017597] LustreError: 285209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7959.037648] LustreError: 285209:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7960.288938] LustreError: 285195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820791 with bad export cookie 13513907168426958349 [ 7960.295716] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7960.301312] LustreError: 285195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7960.620442] Lustre: server umount lustre-MDT0001 complete [ 7968.050271] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 15:06:37 (1786820797) [ 7973.085460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7973.471301] LustreError: 289578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7973.532615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7976.886265] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7978.979503] LustreError: 289579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7984.102150] LustreError: 289578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7984.721430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7985.007278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7988.723826] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7995.568751] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8000.899120] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8006.840818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8007.142661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8012.454652] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8018.079987] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8028.643423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8028.644079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8028.659077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8028.663632] Lustre: Skipped 2 previous similar messages [ 8031.837628] Lustre: server umount lustre-OST0000 complete [ 8036.326085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8036.332414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8036.339083] Lustre: Skipped 1 previous similar message [ 8036.357385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8041.276997] Lustre: server umount lustre-MDT0000 complete [ 8044.001204] LustreError: 291538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8044.020813] LustreError: 291538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8044.239952] LustreError: 289560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820875 with bad export cookie 13513907168426959224 [ 8044.247862] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8044.524225] Lustre: server umount lustre-MDT0001 complete [ 8053.212846] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 8055.650384] Key type lgssc unregistered [ 8055.959406] LNet: 293116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8055.982555] LNetError: 293116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8056.000922] LNet: Removed LNI 192.168.206.127@tcp [ 8056.724665] Key type .llcrypt unregistered [ 8056.726599] Key type ._llcrypt unregistered [ 8079.586527] Key type ._llcrypt registered [ 8079.588829] Key type .llcrypt registered [ 8079.713366] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 8092.139692] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 8093.025745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8093.133538] alg: No test for adler32 (adler32-zlib) [ 8094.164284] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 8094.335976] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 8095.969308] Key type lgssc registered [ 8096.929660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8135.038889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8135.075779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8136.417451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8136.440540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8136.483241] Lustre: lustre-MDT0000: new disk, initializing [ 8136.527120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8136.536965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8139.773458] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8151.568595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8151.724727] Lustre: 296918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8151.815981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8151.825258] Lustre: Skipped 1 previous similar message [ 8151.964760] Lustre: lustre-MDT0001: new disk, initializing [ 8152.111318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8152.173455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8152.190185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8156.306615] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8166.079695] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8170.554933] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8175.557842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8175.728203] Lustre: lustre-OST2710: new disk, initializing [ 8175.730866] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8175.735804] Lustre: 298887:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8175.809869] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8180.548318] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8182.828117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8182.837776] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8182.866682] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8188.658916] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8194.483043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8194.667799] Lustre: lustre-OST03e8: new disk, initializing [ 8194.671203] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8194.676660] Lustre: 299949:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8194.774864] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8200.391410] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8202.830830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8202.849652] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8202.937631] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8206.789542] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8213.569513] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8213.794040] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8217.439361] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8217.619054] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8221.266945] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8221.545989] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8225.759369] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8225.959099] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8233.966959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8233.969964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8233.981737] Lustre: Skipped 3 previous similar messages [ 8233.986723] Lustre: Skipped 2 previous similar messages [ 8233.992863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8236.088663] Lustre: server umount lustre-MDT0000 complete [ 8239.629294] LustreError: 296910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821070 with bad export cookie 9841455644800004071 [ 8239.636968] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8239.647036] LustreError: 296910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8239.959126] Lustre: server umount lustre-MDT0001 complete [ 8254.721280] Lustre: server umount lustre-OST2710 complete [ 8267.457978] Lustre: server umount lustre-OST03e8 complete [ 8284.771473] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 8295.536243] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 8327.187864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8327.393102] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8327.421509] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8327.496198] Lustre: lustre-MDT0000: new disk, initializing [ 8327.543992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8327.559864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8331.360992] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8340.384858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8340.502638] Lustre: 305201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8340.553730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8340.556420] Lustre: Skipped 1 previous similar message [ 8340.648493] Lustre: lustre-MDT0001: new disk, initializing [ 8340.747841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8340.786052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8340.804956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8344.227898] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8352.535686] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8357.548736] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8363.336937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8363.668673] Lustre: lustre-OST0000: new disk, initializing [ 8363.673718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8363.678883] Lustre: 307168:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8363.793492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8368.959392] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8372.336090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8372.367291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8372.529776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8378.755452] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8383.700463] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8383.999289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8389.110524] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8389.318699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8393.191194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8393.192909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8393.210288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8393.210296] Lustre: Skipped 1 previous similar message [ 8393.242694] LustreError: Skipped 1 previous similar message [ 8397.611498] Lustre: server umount lustre-OST0000 complete [ 8402.403608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8402.418955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8402.440114] Lustre: Skipped 1 previous similar message [ 8402.451672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8402.462564] Lustre: Skipped 1 previous similar message [ 8407.546209] Lustre: server umount lustre-MDT0000 complete [ 8408.545606] LustreError: 305206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8408.560909] LustreError: 305206:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8410.782459] LustreError: 305192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821241 with bad export cookie 9841455644800005758 [ 8410.784897] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8410.798032] LustreError: 305192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8411.052575] Lustre: server umount lustre-MDT0001 complete [ 8419.140171] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:14:08 (1786821248) [ 8420.476750] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8422.103850] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:14:11 (1786821251) [ 8425.206317] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 8448.943243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8449.107753] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8449.119804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8449.243326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8453.210446] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8462.348634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8462.510522] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8462.559942] Lustre: 311020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8468.052348] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8492.410501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8492.556412] Lustre: Denying initial registration attempt from nid 192.168.206.127@tcp, specified as failover [ 8492.565792] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8492.574601] LustreError: 312473:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8492.581965] LustreError: 312473:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8492.591443] LustreError: 312473:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8492.643429] Lustre: server umount lustre-OST0000 complete [ 8492.646314] LustreError: 312473:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8498.658373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8498.672933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8498.692346] Lustre: Skipped 1 previous similar message [ 8498.702948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8498.708943] Lustre: Skipped 2 previous similar messages [ 8504.646391] Lustre: server umount lustre-MDT0000 complete [ 8506.850887] LustreError: 311028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8506.861471] LustreError: 311028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8507.817356] LustreError: 311013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821338 with bad export cookie 9841455644800006640 [ 8507.823749] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8507.828622] LustreError: 311013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8508.085323] Lustre: server umount lustre-MDT0001 complete [ 8518.220567] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 8520.770689] Key type lgssc unregistered [ 8521.051597] LNet: 313598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8521.055822] LNetError: 313598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8521.066730] LNet: Removed LNI 192.168.206.127@tcp [ 8521.988804] Key type .llcrypt unregistered [ 8521.991305] Key type ._llcrypt unregistered [ 8533.183134] Key type ._llcrypt registered [ 8533.186114] Key type .llcrypt registered [ 8533.311629] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 8534.378480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8534.540066] alg: No test for adler32 (adler32-zlib) [ 8535.533568] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 8535.815690] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 8537.506180] Key type lgssc registered [ 8538.464057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8580.804596] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 8587.386935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8587.419930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8588.628525] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8588.644686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8588.823186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8592.750668] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8600.895657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8600.960139] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8600.976761] Lustre: 316343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8601.143307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8604.159039] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8612.337642] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8617.470461] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8635.434350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8635.589890] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8635.749710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8641.318731] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8647.704878] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8652.769734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8652.776734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8652.790318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8653.296190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8653.296865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8653.318404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8653.320445] Lustre: Skipped 1 previous similar message [ 8655.763808] Lustre: server umount lustre-OST0000 complete [ 8662.498297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8662.515351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8662.530901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8665.622959] Lustre: server umount lustre-MDT0000 complete [ 8669.300703] LustreError: 316334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821500 with bad export cookie 3077316270381650097 [ 8669.305527] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8669.312063] LustreError: 316334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8669.552429] Lustre: server umount lustre-MDT0001 complete [ 8680.376537] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 8682.963324] Key type lgssc unregistered [ 8683.275804] LNet: 319918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8683.281379] LNetError: 319918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8683.294652] LNet: Removed LNI 192.168.206.127@tcp [ 8684.061497] Key type .llcrypt unregistered [ 8684.067090] Key type ._llcrypt unregistered [ 8704.159401] Key type ._llcrypt registered [ 8704.161690] Key type .llcrypt registered [ 8704.269609] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 8705.642075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8705.680656] alg: No test for adler32 (adler32-zlib) [ 8706.869356] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 8707.109387] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 8708.768178] Key type lgssc registered [ 8709.700837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8718.022725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8718.044895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8719.412202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8723.457342] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8729.570239] LustreError: 321315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8729.596061] LustreError: 321315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8730.760657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8731.061896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8735.234155] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8741.639906] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8746.717879] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8755.056918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8755.384843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8761.509841] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8768.801941] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8780.261300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8780.263996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8780.269088] Lustre: Skipped 1 previous similar message [ 8780.279134] Lustre: Skipped 2 previous similar messages [ 8781.953947] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.27@tcp (stopping) [ 8784.316057] Lustre: server umount lustre-MDT0000 complete [ 8785.393258] LustreError: 321315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8785.414241] LustreError: 321315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8787.050847] LustreError: 321310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8787.929589] LustreError: 321295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821618 with bad export cookie 11880416572167939016 [ 8787.935443] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8787.947896] LustreError: 321295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8790.497309] LustreError: 323131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8790.502914] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8790.538550] Lustre: Skipped 1 previous similar message [ 8790.550892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8794.542897] Lustre: server umount lustre-MDT0001 complete [ 8800.528547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8808.291451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8813.221042] LustreError: 323147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8813.234069] LustreError: 323147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8813.289131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8813.351741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8817.172466] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8822.498483] LustreError: 323147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8822.502144] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8822.515576] LustreError: 323147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8824.607564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8825.031621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8828.501662] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8830.436907] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8834.327468] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8835.719841] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8840.507153] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8842.051681] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8849.379643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8849.389373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8849.402814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8849.413676] Lustre: Skipped 2 previous similar messages [ 8850.402383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8852.589242] Lustre: server umount lustre-OST0000 complete [ 8860.643427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8860.651493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8860.659717] Lustre: Skipped 1 previous similar message [ 8860.664173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8860.668486] Lustre: Skipped 2 previous similar messages [ 8870.370810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8870.386859] Lustre: Skipped 5 previous similar messages [ 8871.392122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8871.887593] Lustre: server umount lustre-MDT0000 complete [ 8875.490585] LustreError: 324905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8875.522048] LustreError: 324905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8875.744285] LustreError: 321295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821706 with bad export cookie 11880416572167946163 [ 8875.763183] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8875.769055] LustreError: 321295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8876.128289] Lustre: server umount lustre-MDT0001 complete [ 8887.566049] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 8890.215975] Key type lgssc unregistered [ 8890.563933] LNet: 327582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8890.577317] LNetError: 327582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8890.601127] LNet: Removed LNI 192.168.206.127@tcp [ 8891.336380] Key type .llcrypt unregistered [ 8891.337934] Key type ._llcrypt unregistered [ 8917.048541] Key type ._llcrypt registered [ 8917.052103] Key type .llcrypt registered [ 8917.239388] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 8918.103028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8918.133595] alg: No test for adler32 (adler32-zlib) [ 8919.241331] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 8919.486995] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 8921.192162] Key type lgssc registered [ 8923.083818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8930.555915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8930.586342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8932.065777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8935.967927] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8941.540952] LustreError: 329002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8941.560169] LustreError: 329002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8943.756623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8944.090088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8947.875336] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8955.413679] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8960.590359] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8964.579145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8964.588750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8964.601761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8966.625987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8966.627811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8966.658867] Lustre: Skipped 1 previous similar message [ 8968.739471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8968.747899] Lustre: Skipped 1 previous similar message [ 8969.458253] Lustre: server umount lustre-MDT0000 complete [ 8972.678223] LustreError: 328983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821803 with bad export cookie 14863313628478413665 [ 8972.689373] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8972.960855] Lustre: server umount lustre-MDT0001 complete [ 8987.747579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8988.112815] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8988.145312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8988.386815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8992.523990] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9001.182499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9001.297822] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9001.338771] Lustre: 331587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9001.517141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9005.769473] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9013.193580] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9018.552454] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9027.962083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9028.226226] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9028.243588] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9028.254629] LustreError: 333471:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9028.261836] LustreError: 333471:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9028.274834] LustreError: 333471:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9028.327359] Lustre: server umount lustre-OST0000 complete [ 9028.331788] LustreError: 333471:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9037.739441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9037.891809] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9038.054387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9044.343516] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9047.661023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9051.306092] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9059.212991] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9059.327745] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9059.401754] Lustre: lustre-OST0001: new disk, initializing [ 9059.408304] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9059.414332] Lustre: 335011:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9059.556634] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9066.559752] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9069.699553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9069.711635] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9069.875862] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9075.514565] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9080.289458] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9080.303109] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9080.305025] LustreError: Skipped 1 previous similar message [ 9080.306356] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9080.332512] Lustre: Skipped 1 previous similar message [ 9084.612910] Lustre: server umount lustre-OST0001 complete [ 9085.416315] LustreError: 333939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9085.438600] LustreError: 333939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9088.482305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9088.488916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9088.501186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9088.505559] Lustre: Skipped 1 previous similar message [ 9090.542932] LustreError: 333936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9090.565037] LustreError: 333936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9095.658790] LustreError: 335488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9095.673567] LustreError: 335488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9100.774901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9100.775321] LustreError: 333936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9100.789534] Lustre: Skipped 6 previous similar messages [ 9100.823560] LustreError: 333936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9102.305137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9102.533602] Lustre: server umount lustre-OST0000 complete [ 9108.961313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9108.970366] LustreError: Skipped 1 previous similar message [ 9108.973591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9108.979928] Lustre: Skipped 1 previous similar message [ 9114.650297] Lustre: server umount lustre-MDT0000 complete [ 9116.130891] LustreError: 331599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9116.143903] LustreError: 331599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9118.090927] LustreError: 331579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821948 with bad export cookie 14863313628478414218 [ 9118.094355] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9118.101219] LustreError: 331579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9118.638717] Lustre: server umount lustre-MDT0001 complete [ 9130.361151] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 9133.241360] Key type lgssc unregistered [ 9133.519815] LNet: 336976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9133.534118] LNetError: 336976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9133.554369] LNet: Removed LNI 192.168.206.127@tcp [ 9134.450411] Key type .llcrypt unregistered [ 9134.459836] Key type ._llcrypt unregistered [ 9206.099360] Key type ._llcrypt registered [ 9206.102405] Key type .llcrypt registered [ 9206.198171] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 9218.887958] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 9219.853349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9219.871810] alg: No test for adler32 (adler32-zlib) [ 9220.953124] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 9221.168746] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 9222.824192] Key type lgssc registered [ 9223.839763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9254.210993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9254.238439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9255.467157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9255.509417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9255.621897] Lustre: lustre-MDT0000: new disk, initializing [ 9255.746154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9255.795144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9260.064464] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9269.187146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9269.347903] Lustre: 341890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9269.438843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9269.446435] Lustre: Skipped 1 previous similar message [ 9269.499686] Lustre: lustre-MDT0001: new disk, initializing [ 9269.609111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9269.647577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9269.673464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9273.275488] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9282.086734] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9287.390250] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9293.669924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9293.920224] Lustre: lustre-OST0000: new disk, initializing [ 9293.924622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9293.931704] Lustre: 343861:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9294.047652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9299.601709] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9300.037788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9300.051329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9300.164345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9307.134251] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9311.779835] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9312.046889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9316.109159] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9316.311433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9320.929785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9320.941711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9320.977809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9320.988629] Lustre: Skipped 1 previous similar message [ 9324.332521] Lustre: server umount lustre-OST0000 complete [ 9331.171842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9331.172970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9331.185682] Lustre: Skipped 1 previous similar message [ 9331.193135] LustreError: Skipped 1 previous similar message [ 9331.194595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9331.218732] Lustre: Skipped 1 previous similar message [ 9333.919921] Lustre: server umount lustre-MDT0000 complete [ 9337.293823] LustreError: 341881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822168 with bad export cookie 16811346788201147420 [ 9337.297244] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9337.310268] LustreError: 341881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9337.584655] Lustre: server umount lustre-MDT0001 complete [ 9344.976650] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:29:34 (1786822174) [ 9352.048859] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:29:41 (1786822181) [ 9359.290796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9359.704937] LustreError: 346613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9359.723920] LustreError: 346613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9359.780571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9364.162098] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9364.964241] LustreError: 346614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9370.095069] LustreError: 346614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9372.026694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9372.471459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9377.039306] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9384.268676] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9389.118268] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9394.813572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9395.122488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9400.006280] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9405.943541] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9412.655394] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9414.146918] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9415.439794] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9416.936336] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9418.583876] Lustre: Failing over lustre-MDT0000 [ 9418.725197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9418.731878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9418.743347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9418.842988] Lustre: server umount lustre-MDT0000 complete [ 9419.748769] LustreError: 346614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9419.767452] LustreError: 346614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9424.868016] LustreError: 349102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9424.883984] LustreError: 349102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9433.193094] LustreError: 346609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9433.210915] LustreError: 346609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9435.953214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9436.047762] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9436.276229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9436.345806] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9438.317050] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9439.987990] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9441.804242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9441.848213] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9441.889329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9447.564792] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9448.773523] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9451.729603] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9453.273325] Lustre: Failing over lustre-MDT0000 [ 9453.498429] Lustre: server umount lustre-MDT0000 complete [ 9453.700780] LustreError: 346609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9453.716829] LustreError: 346609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9457.122245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9457.125703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9457.141163] Lustre: Skipped 4 previous similar messages [ 9470.530377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9470.609671] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9470.773598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9470.829239] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9474.187869] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9475.181886] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9476.071486] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9476.075773] Lustre: Skipped 2 previous similar messages [ 9476.112861] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9476.172238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9481.713795] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9482.865960] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9491.427124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9491.431696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9491.433587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9491.448545] Lustre: Skipped 2 previous similar messages [ 9493.825956] Lustre: server umount lustre-MDT0000 complete [ 9496.552842] LustreError: 346614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9496.566687] LustreError: 346614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 9496.776549] LustreError: 346595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822327 with bad export cookie 16811346788201150129 [ 9496.780831] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9496.786592] LustreError: 346595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9497.072705] Lustre: server umount lustre-MDT0001 complete [ 9510.993744] Lustre: server umount lustre-OST0000 complete [ 9523.654385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9527.282320] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9533.592211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9536.879845] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9543.017446] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9548.540467] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9554.941060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9555.463468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9555.470606] Lustre: Skipped 2 previous similar messages [ 9561.738231] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9568.819548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9570.978922] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9578.977579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9578.987575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9578.996209] Lustre: Skipped 1 previous similar message [ 9578.997720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9579.013311] Lustre: Skipped 3 previous similar messages [ 9584.315275] Lustre: server umount lustre-OST0000 complete [ 9589.221654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9589.224598] Lustre: Skipped 3 previous similar messages [ 9594.274492] Lustre: server umount lustre-MDT0000 complete [ 9594.337370] LustreError: 354041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9594.350345] LustreError: 354041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9597.497228] LustreError: 352927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822428 with bad export cookie 16811346788201150850 [ 9597.504042] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9597.508824] LustreError: 352927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9597.837161] Lustre: server umount lustre-MDT0001 complete [ 9607.237940] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 9609.507667] Key type lgssc unregistered [ 9609.773924] LNet: 356481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9609.783520] LNetError: 356481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9609.795495] LNet: Removed LNI 192.168.206.127@tcp [ 9610.448333] Key type .llcrypt unregistered [ 9610.450182] Key type ._llcrypt unregistered [ 9631.894551] Key type ._llcrypt registered [ 9631.898140] Key type .llcrypt registered [ 9632.004266] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 9632.713498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9632.783872] alg: No test for adler32 (adler32-zlib) [ 9633.812659] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 9634.030587] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 9635.680153] Key type lgssc registered [ 9636.726449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9643.643628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9643.665901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9645.041095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9649.147458] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9654.245661] LustreError: 357976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9654.263372] LustreError: 357976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9656.684837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9657.087563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9661.210808] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9667.592378] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9672.162572] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9678.166443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9678.459757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9684.183619] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9689.736887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9690.278358] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9700.324511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9700.327596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9700.339299] Lustre: Skipped 2 previous similar messages [ 9700.350025] Lustre: Skipped 2 previous similar messages [ 9705.444722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9705.448408] Lustre: Skipped 2 previous similar messages [ 9706.017094] Lustre: server umount lustre-MDT0000 complete [ 9710.048134] LustreError: 357957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822540 with bad export cookie 17125830752984195138 [ 9710.050261] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9710.056948] LustreError: 357957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9710.366782] Lustre: server umount lustre-MDT0001 complete [ 9724.936484] Lustre: server umount lustre-OST0000 complete [ 9736.944569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9737.423304] LustreError: 361687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9737.481788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9740.956261] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9742.817277] LustreError: 361688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9747.015127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9747.237824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9750.591489] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9757.196620] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9762.755521] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9769.257525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9769.552540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9774.706920] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9782.724523] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9783.917976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9799.651279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9799.661962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9799.673447] Lustre: Skipped 2 previous similar messages [ 9799.693322] Lustre: Skipped 2 previous similar messages [ 9804.775671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9804.784820] Lustre: Skipped 2 previous similar messages [ 9805.552717] Lustre: server umount lustre-MDT0000 complete [ 9809.045550] LustreError: 361669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822639 with bad export cookie 17125830752984196251 [ 9809.046387] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9809.054660] LustreError: 361669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9809.455856] Lustre: server umount lustre-MDT0001 complete [ 9823.178348] Lustre: server umount lustre-OST0000 complete [ 9838.762134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9839.384337] LustreError: 365601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9839.453409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9843.876670] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9844.707844] LustreError: 365602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9852.631880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9856.928774] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9863.618578] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9869.766856] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9875.770269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9876.097662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9876.103935] Lustre: Skipped 1 previous similar message [ 9882.068583] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9884.281248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9889.193592] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9894.884179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9894.892046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9894.892440] LustreError: Skipped 1 previous similar message [ 9894.894336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9894.922662] Lustre: Skipped 1 previous similar message [ 9899.280193] Lustre: server umount lustre-OST0000 complete [ 9904.096842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9904.102906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9904.112224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9904.119291] Lustre: Skipped 1 previous similar message [ 9917.409535] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9917.507991] LustreError: 368379:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000048b609f4) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 9917.643759] Lustre: server umount lustre-MDT0000 complete [ 9919.462760] LustreError: 366329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9919.481495] LustreError: 366329:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9921.145943] LustreError: 365582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822752 with bad export cookie 17125830752984197273 [ 9921.166142] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9921.571812] Lustre: server umount lustre-MDT0001 complete [ 9931.810557] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 9934.549325] Key type lgssc unregistered [ 9934.789516] LNet: 369138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9934.801150] LNetError: 369138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9934.821583] LNet: Removed LNI 192.168.206.127@tcp [ 9935.660799] Key type .llcrypt unregistered [ 9935.662924] Key type ._llcrypt unregistered [ 9961.959880] Key type ._llcrypt registered [ 9961.964049] Key type .llcrypt registered [ 9962.111585] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 9963.144665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9963.251130] alg: No test for adler32 (adler32-zlib) [ 9964.405201] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 9964.674971] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 9966.424162] Key type lgssc registered [ 9967.622980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9974.799325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9974.824578] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9974.838614] LustreError: 370578:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9974.856476] LustreError: 370578:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9974.864696] LustreError: 370578:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9974.872882] LustreError: 370578:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9974.878902] LustreError: 370578:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9981.376066] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9981.382472] LustreError: 370975:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9981.396610] LustreError: 370975:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9981.403344] LustreError: 370975:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9981.407863] LustreError: 370975:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9981.414925] LustreError: 370975:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9998.322123] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [10001.080972] Key type lgssc unregistered [10001.406552] LNet: 371983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10001.421284] LNetError: 371983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10001.444923] LNet: Removed LNI 192.168.206.127@tcp [10002.219866] Key type .llcrypt unregistered [10002.226658] Key type ._llcrypt unregistered [10020.195750] Key type ._llcrypt registered [10020.197680] Key type .llcrypt registered [10020.327760] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [10037.019642] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [10038.334113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10038.507924] alg: No test for adler32 (adler32-zlib) [10039.612695] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [10039.964893] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [10041.792463] Key type lgssc registered [10043.210655] Lustre: Echo OBD driver; http://www.lustre.org/ [10075.826813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10075.854939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10077.164597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10077.190827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10077.246270] Lustre: lustre-MDT0000: new disk, initializing [10077.308772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10077.328859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10081.117405] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10091.276580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10091.408893] Lustre: 375143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10091.513698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10091.527385] Lustre: Skipped 1 previous similar message [10091.584271] Lustre: lustre-MDT0001: new disk, initializing [10091.655077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10091.704925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10091.720358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10096.411679] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10105.542666] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10110.952462] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10117.468455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10117.853393] Lustre: lustre-OST0000: new disk, initializing [10117.859857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10117.868274] Lustre: 377110:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10118.033227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10123.664389] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10126.478710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10126.493996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10126.640754] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10134.239851] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10139.366331] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10139.547305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10143.883180] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10144.149649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10147.298085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10147.301381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10147.304196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10147.320023] Lustre: Skipped 1 previous similar message [10152.374702] Lustre: server umount lustre-OST0000 complete [10157.544692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10157.546706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10157.561943] Lustre: Skipped 1 previous similar message [10157.565097] Lustre: Skipped 1 previous similar message [10162.657174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10162.668075] Lustre: Skipped 2 previous similar messages [10167.778945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10170.337405] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10170.525628] Lustre: server umount lustre-MDT0000 complete [10172.898533] LustreError: 375155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10172.914747] LustreError: 375155:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10173.736180] LustreError: 375939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823004 with bad export cookie 13318266223435406453 [10173.751157] LustreError: 375939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10173.753522] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10174.061986] Lustre: server umount lustre-MDT0001 complete [10182.444524] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10184.182826] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:43:33 (1786823013) [10190.457863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10190.940069] LustreError: 379572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10191.022638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10195.388962] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10196.451571] LustreError: 379573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10196.481276] LustreError: 379573:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10200.546976] LustreError: 379572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10203.751808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10204.003466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10208.984992] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10215.768846] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10221.430095] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10228.869898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10229.480569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10236.332827] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10243.561155] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10249.267432] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10249.410054] Lustre: lustre-OST0001: new disk, initializing [10249.413363] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10249.420582] Lustre: 382409:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10249.486735] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10255.412744] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10258.989805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10258.999163] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10259.086390] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10264.805980] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10269.156020] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10269.162175] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10269.173425] Lustre: Skipped 1 previous similar message [10269.181450] Lustre: Skipped 2 previous similar messages [10274.632782] Lustre: server umount lustre-OST0001 complete [10276.984459] LustreError: 381394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10289.642293] LustreError: 382372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10289.665148] LustreError: 382372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10292.193099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10292.203429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10292.216909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10292.221571] Lustre: Skipped 3 previous similar messages [10297.218512] Lustre: server umount lustre-OST0000 complete [10301.410348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10301.423825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10301.440400] Lustre: Skipped 1 previous similar message [10306.706993] Lustre: server umount lustre-MDT0000 complete [10309.989047] LustreError: 379553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823140 with bad export cookie 13318266223435407335 [10310.001409] LustreError: 379553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10310.005948] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10316.402124] Lustre: server umount lustre-MDT0001 complete [10326.585155] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [10329.254710] Key type lgssc unregistered [10329.521077] LNet: 384396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10329.526598] LNetError: 384396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10329.566616] LNet: Removed LNI 192.168.206.127@tcp [10330.469875] Key type .llcrypt unregistered [10330.472482] Key type ._llcrypt unregistered [10380.336618] Key type ._llcrypt registered [10380.339708] Key type .llcrypt registered [10380.445446] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [10381.613404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10381.626402] alg: No test for adler32 (adler32-zlib) [10382.646795] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [10382.934321] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [10384.632442] Key type lgssc registered [10385.836764] Lustre: Echo OBD driver; http://www.lustre.org/ [10393.187386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10393.218611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10394.500523] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10394.531644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10394.666405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10398.679536] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10406.725526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10406.854897] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10406.881858] Lustre: 387190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10407.142297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10411.035964] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10418.886420] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10426.419420] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10433.008959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10433.020455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10433.038237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10438.625370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10438.629613] Lustre: Skipped 3 previous similar messages [10443.232133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10443.412095] Lustre: server umount lustre-MDT0000 complete [10447.217553] LustreError: 387942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823278 with bad export cookie 5796564531588167132 [10447.229279] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10447.707584] Lustre: server umount lustre-MDT0001 complete [10457.615386] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:48:06 (1786823286) [10464.331835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10464.822646] LustreError: 389703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10464.846788] LustreError: 389703:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10464.912148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10470.380095] LustreError: 389704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10470.706680] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10475.489329] LustreError: 389703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10480.371963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10480.609353] LustreError: 389704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10480.801026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10484.921533] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10492.396036] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10497.743528] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10503.843035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10503.961417] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10504.118202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10509.462957] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10515.904859] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10522.938497] Lustre: Permanently deactivating lustre-OST0000 [10522.971754] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10532.445374] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10533.844734] LustreError: 392738:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10533.863318] LustreError: 392738:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10533.871666] LustreError: 392738:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10536.817580] LustreError: 392787:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10536.834878] LustreError: 392787:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10544.628583] Lustre: server umount lustre-OST0000 complete [10547.279612] LustreError: 393036:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10547.288595] LustreError: 393036:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10552.803627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10552.812149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10552.828634] Lustre: Skipped 1 previous similar message [10552.833831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10558.434182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10558.443743] Lustre: Skipped 3 previous similar messages [10559.509477] Lustre: server umount lustre-MDT0000 complete [10562.915443] LustreError: 389684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823393 with bad export cookie 5796564531588167713 [10562.922892] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10562.929505] LustreError: 389684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10572.396698] Lustre: server umount lustre-MDT0001 complete [10580.005044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10584.672657] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10587.004031] LustreError: 394195:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10593.039652] Lustre: 394534:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.127@tcp [10595.077842] Lustre: server umount MGS complete [10608.018802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10608.283196] LustreError: 395258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10608.299545] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10608.302443] Lustre: Skipped 1 previous similar message [10608.334388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10612.464565] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10618.850160] LustreError: 395259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10618.870879] LustreError: 395259:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10620.260470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10620.535502] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10624.443229] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10631.294451] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10636.361770] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10638.343532] Lustre: Permanently reactivating lustre-OST0000 [10638.356941] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10638.363374] Lustre: Skipped 2 previous similar messages [10665.444647] LustreError: 397039:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10672.536344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10672.810382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10672.817910] Lustre: Skipped 1 previous similar message [10679.002545] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10685.960364] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10690.529888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10690.538461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10690.556887] Lustre: Skipped 1 previous similar message [10690.562352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10696.681772] Lustre: server umount lustre-OST0000 complete [10701.793619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10701.797199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10701.807844] Lustre: Skipped 2 previous similar messages [10701.815461] Lustre: Skipped 3 previous similar messages [10706.753724] Lustre: server umount lustre-MDT0000 complete [10706.913348] LustreError: 397571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10706.936941] LustreError: 397571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10711.035524] LustreError: 395239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823541 with bad export cookie 5796564531588168756 [10711.037764] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10711.044764] LustreError: 395239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10711.438188] Lustre: server umount lustre-MDT0001 complete [10722.824285] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [10725.480438] Key type lgssc unregistered [10725.726822] LNet: 399674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10725.733422] LNetError: 399674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10725.755146] LNet: Removed LNI 192.168.206.127@tcp [10726.733352] Key type .llcrypt unregistered [10726.736249] Key type ._llcrypt unregistered [10744.928063] Key type ._llcrypt registered [10744.929803] Key type .llcrypt registered [10745.082641] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [10762.044468] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [10763.403990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10763.594589] alg: No test for adler32 (adler32-zlib) [10765.256664] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [10765.868769] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [10767.680204] Key type lgssc registered [10770.192750] Lustre: Echo OBD driver; http://www.lustre.org/ [10807.372173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10807.400314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10808.663538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10808.706946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10808.797408] Lustre: lustre-MDT0000: new disk, initializing [10808.896458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10808.910937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10812.581463] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10822.067169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10822.234767] Lustre: 402832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10822.295224] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10822.298329] Lustre: Skipped 1 previous similar message [10822.370772] Lustre: lustre-MDT0001: new disk, initializing [10822.475110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10822.521807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10822.532393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10826.451405] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10835.897671] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10841.942600] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10849.744270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10850.107280] Lustre: lustre-OST0000: new disk, initializing [10850.112484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10850.130172] Lustre: 404801:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10850.295609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10857.563655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10857.577468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10857.717835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10858.002978] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10867.914214] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10872.257729] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10872.601598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10878.375826] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10878.592817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10883.556890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10883.558600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10883.593434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10883.601819] Lustre: Skipped 1 previous similar message [10888.697029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10888.702521] Lustre: Skipped 1 previous similar message [10893.795820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10893.801400] Lustre: Skipped 1 previous similar message [10895.328141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10895.458469] Lustre: server umount lustre-OST0000 complete [10898.913793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10898.914842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10898.927229] Lustre: Skipped 2 previous similar messages [10898.931883] Lustre: Skipped 1 previous similar message [10904.040642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10904.045380] Lustre: Skipped 1 previous similar message [10913.248205] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10913.394347] Lustre: server umount lustre-MDT0000 complete [10917.023953] LustreError: 402824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823747 with bad export cookie 18290403604623370219 [10917.032950] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10917.305140] Lustre: server umount lustre-MDT0001 complete [10926.104590] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10927.811656] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:55:57 (1786823757) [10934.364644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10934.859338] LustreError: 407263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10934.881315] LustreError: 407263:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10934.945701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10938.985203] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10940.389978] LustreError: 407264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10944.482746] LustreError: 407263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10947.054224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10947.296738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10951.314531] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10957.835984] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10963.338560] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10968.871176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10969.208300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10976.839721] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10984.810547] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10987.208535] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10989.675526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10995.169965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10995.175700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10995.190104] LustreError: Skipped 1 previous similar message [10995.195659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10995.195667] Lustre: Skipped 2 previous similar messages [10995.236020] Lustre: Skipped 1 previous similar message [11000.292333] LustreError: 409563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11000.305510] LustreError: 409563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11000.579062] Lustre: server umount lustre-OST0000 complete [11005.416118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11005.441627] Lustre: Skipped 1 previous similar message [11015.655536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11015.668541] Lustre: Skipped 6 previous similar messages [11019.232246] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11019.357284] Lustre: server umount lustre-MDT0000 complete [11020.769191] LustreError: 407263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11020.788943] LustreError: 407263:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11022.579911] LustreError: 407244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823853 with bad export cookie 18290403604623371101 [11022.596498] LustreError: 407244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11022.625133] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11023.001516] Lustre: server umount lustre-MDT0001 complete [11032.978609] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [11035.439822] Key type lgssc unregistered [11035.797263] LNet: 410993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11035.805140] LNetError: 410993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11035.825211] LNet: Removed LNI 192.168.206.127@tcp [11036.732350] Key type .llcrypt unregistered [11036.734338] Key type ._llcrypt unregistered [11062.199959] Key type ._llcrypt registered [11062.203511] Key type .llcrypt registered [11062.290197] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [11062.529679] Key type .llcrypt unregistered [11062.535279] Key type ._llcrypt unregistered [11080.162630] Key type ._llcrypt registered [11080.165594] Key type .llcrypt registered [11080.306702] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [11081.572525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11081.684264] alg: No test for adler32 (adler32-zlib) [11082.802404] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [11083.046971] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [11084.728290] Key type lgssc registered [11085.703821] Lustre: Echo OBD driver; http://www.lustre.org/ [11092.631360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11092.646714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11094.038047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11098.077241] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11103.203066] LustreError: 413211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11103.216658] LustreError: 413211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11106.457565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11106.763133] LustreError: 413212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11106.799139] LustreError: 413212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11106.808225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11111.910357] LustreError: 413207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11111.936417] LustreError: 413207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11113.324202] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11117.028821] LustreError: 413208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11117.047315] LustreError: 413208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11119.093051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11120.694281] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11124.199633] LustreError: 413208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11124.216383] LustreError: 413208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11128.180106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11128.532048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11133.012872] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11143.247296] Lustre: server umount lustre-OST0000 complete [11148.782779] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11148.795854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11148.821643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11154.401574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11154.416244] Lustre: Skipped 3 previous similar messages [11157.611485] Lustre: server umount lustre-MDT0000 complete [11159.521901] LustreError: 413212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11159.537992] LustreError: 413212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11160.420760] Lustre: 412522:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786823975/real 1786823975] req@00000000651720a1 x1873620671423616/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786823991 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11160.456692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11160.483548] Lustre: Skipped 1 previous similar message [11161.121141] LustreError: 413192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823991 with bad export cookie 4355140458560054781 [11161.132943] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11161.135396] LustreError: 413192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11161.770741] Lustre: server umount lustre-MDT0001 complete [11172.886742] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [11175.677214] Key type lgssc unregistered [11176.086268] LNet: 416363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11176.096881] LNetError: 416363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11176.116456] LNet: Removed LNI 192.168.206.127@tcp [11177.184245] Key type .llcrypt unregistered [11177.186499] Key type ._llcrypt unregistered [11201.907868] Key type ._llcrypt registered [11201.914091] Key type .llcrypt registered [11202.061956] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [11203.320756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11203.522964] alg: No test for adler32 (adler32-zlib) [11204.615737] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [11204.919795] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [11206.600810] Key type lgssc registered [11207.746012] Lustre: Echo OBD driver; http://www.lustre.org/ [11215.842728] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11215.854710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11231.392558] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11236.512478] LustreError: 417752:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.127@tcp: failed processing log, type 4: rc = -110 [11263.201932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11269.728155] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11278.110814] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11285.533322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11286.146430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11286.227204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11290.386814] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11291.621699] LustreError: 417778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11291.651446] LustreError: 417778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11296.743952] LustreError: 417777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11297.769963] LustreError: 417778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11297.804039] LustreError: 417778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11299.874745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11300.384926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11304.797846] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11312.129301] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11317.567572] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11325.910470] Lustre: server umount lustre-OST0000 complete [11331.043877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11331.046963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11331.051715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11331.068067] Lustre: Skipped 1 previous similar message [11336.165319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11336.171777] Lustre: Skipped 2 previous similar messages [11340.318892] Lustre: server umount lustre-MDT0000 complete [11341.285902] LustreError: 418804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11341.286764] Lustre: 417091:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786824156/real 1786824156] req@00000000e9232681 x1873620799353216/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786824172 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11341.298727] LustreError: 418804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11341.329611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11343.817887] LustreError: 420291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824174 with bad export cookie 2976936444652628263 [11343.822484] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11343.829309] LustreError: 420291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11344.105266] Lustre: server umount lustre-MDT0001 complete [11355.499394] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [11358.253819] Key type lgssc unregistered [11358.466549] LNet: 421335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11358.470827] LNetError: 421335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11358.486746] LNet: Removed LNI 192.168.206.127@tcp [11359.208818] Key type .llcrypt unregistered [11359.212954] Key type ._llcrypt unregistered [11391.911709] Key type ._llcrypt registered [11391.917529] Key type .llcrypt registered [11392.191789] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [11394.151385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11394.219044] alg: No test for adler32 (adler32-zlib) [11395.518283] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [11395.917904] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [11397.664273] Key type lgssc registered [11398.669548] Lustre: Echo OBD driver; http://www.lustre.org/ [11407.256159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11407.291635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11408.885809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11414.101324] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11419.112296] LustreError: 422755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11419.138949] LustreError: 422755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11422.249457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11422.605462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11426.975597] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11434.599974] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11441.236752] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11448.339793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11448.806484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11453.965446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11456.280303] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11464.468676] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11468.769065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11468.778109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11468.804170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11471.522346] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.27@tcp (stopping) [11472.364635] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11474.812813] Lustre: server umount lustre-MDT0000 complete [11476.595530] LustreError: 422750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11477.486621] LustreError: 422736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.27@tcp arrived at 1786824308 with bad export cookie 10624873803133330893 [11477.512057] LustreError: 422736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11479.422979] LustreError: 422734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824310 with bad export cookie 10624873803133330004 [11479.441466] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11479.442357] LustreError: 422734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11482.593683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11482.605325] LustreError: 422755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11482.612702] Lustre: Skipped 1 previous similar message [11482.613775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11482.615291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11482.623728] LustreError: 422755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11482.649179] Lustre: Skipped 3 previous similar messages [11484.641316] LustreError: 422754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11484.654822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11484.660676] LustreError: 422754:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11485.701477] Lustre: server umount lustre-OST0000 complete [11489.768014] LustreError: 422755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11489.791542] LustreError: 422755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11492.318086] Lustre: server umount lustre-MDT0001 complete [11504.113554] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [11507.458752] Key type lgssc unregistered [11507.771355] LNet: 426439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11507.779120] LNetError: 426439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11507.790743] LNet: Removed LNI 192.168.206.127@tcp [11508.709585] Key type .llcrypt unregistered [11508.711960] Key type ._llcrypt unregistered [11532.653380] Key type ._llcrypt registered [11532.656464] Key type .llcrypt registered [11532.777652] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [11533.814910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11533.833468] alg: No test for adler32 (adler32-zlib) [11534.833570] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [11535.033131] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [11536.712155] Key type lgssc registered [11538.142476] Lustre: Echo OBD driver; http://www.lustre.org/ [11547.575620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11547.588698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11549.095769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11553.637475] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11558.372963] LustreError: 427873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11558.387925] LustreError: 427873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11562.472434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11563.075876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11567.892716] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11576.102786] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11582.400952] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11589.004312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11589.272777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11593.409538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11595.345435] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11602.614837] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11609.069868] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11609.071523] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11609.083488] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11609.093392] Lustre: Skipped 1 previous similar message [11609.106664] Lustre: Skipped 1 previous similar message [11609.711599] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.27@tcp (stopping) [11612.529225] Lustre: server umount lustre-MDT0001 complete [11614.187635] LustreError: 427874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11614.215415] LustreError: 427874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11619.297179] LustreError: 427873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11619.299081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11619.299112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11619.316662] LustreError: 427873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11619.350549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11624.419523] LustreError: 428601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11624.421449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11625.208299] Lustre: server umount lustre-OST0000 complete [11629.550180] LustreError: 427873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11630.655761] Lustre: server umount lustre-MDT0000 complete [11649.643144] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [11652.779700] Key type lgssc unregistered [11653.034822] LNet: 431562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11653.041741] LNetError: 431562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11653.066207] LNet: Removed LNI 192.168.206.127@tcp [11653.791261] Key type .llcrypt unregistered [11653.794785] Key type ._llcrypt unregistered [11678.958466] Key type ._llcrypt registered [11678.961097] Key type .llcrypt registered [11679.058661] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [11692.951543] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [11693.976985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11694.039884] alg: No test for adler32 (adler32-zlib) [11695.202770] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [11695.459115] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [11697.128804] Key type lgssc registered [11698.216645] Lustre: Echo OBD driver; http://www.lustre.org/ [11733.609097] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [11749.609862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11749.661958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11751.044850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11751.093354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11751.234882] Lustre: lustre-MDT0000: new disk, initializing [11751.385260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11751.435256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11756.663887] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11773.271698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11773.499658] Lustre: lustre-OST0000: new disk, initializing [11773.503807] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11773.507911] Lustre: Skipped 1 previous similar message [11773.524869] Lustre: 436599:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11773.619085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11780.254889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11780.262803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11780.318738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11780.381667] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11791.068119] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11809.357191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11809.461617] Lustre: 435581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11809.615276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11809.784121] Lustre: lustre-MDT0001: new disk, initializing [11809.942914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11809.987642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11810.001524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11814.084733] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11823.244629] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11823.375221] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11827.038351] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11827.236311] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11833.552547] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.27@tcp (stopping) [11834.343409] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11834.344199] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11834.347644] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11834.391910] Lustre: Skipped 1 previous similar message [11838.578492] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.27@tcp (stopping) [11838.804535] Lustre: server umount lustre-MDT0001 complete [11844.585113] LustreError: 435589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11844.605226] LustreError: 435589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11849.698916] LustreError: 435593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11849.721591] LustreError: 435593:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11854.820277] LustreError: 435589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11854.835376] LustreError: 435589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11857.381020] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11857.390701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11857.403991] Lustre: Skipped 1 previous similar message [11857.415912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11859.942973] LustreError: 435593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11865.057721] LustreError: 435592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11865.058022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11865.106760] Lustre: Skipped 1 previous similar message [11871.200136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11871.378098] Lustre: server umount lustre-OST0000 complete [11880.930398] Lustre: server umount lustre-MDT0000 complete [11895.153578] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [11898.179336] Key type lgssc unregistered [11898.502789] LNet: 440353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11898.510801] LNetError: 440353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11898.529471] LNet: Removed LNI 192.168.206.127@tcp [11899.568865] Key type .llcrypt unregistered [11899.574877] Key type ._llcrypt unregistered [11942.066923] Key type ._llcrypt registered [11942.073560] Key type .llcrypt registered [11942.258090] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [11942.508770] Key type .llcrypt unregistered [11942.514599] Key type ._llcrypt unregistered [11958.973726] Key type ._llcrypt registered [11958.977321] Key type .llcrypt registered [11959.087663] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [11960.055646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11960.125866] alg: No test for adler32 (adler32-zlib) [11961.273677] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [11961.476748] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [11963.120150] Key type lgssc registered [11964.088269] Lustre: Echo OBD driver; http://www.lustre.org/ [11988.803794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11988.816132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11990.104343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11990.153387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11990.297585] Lustre: lustre-MDT0000: new disk, initializing [11990.413551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11990.449494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11994.477305] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12003.891527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12003.991332] Lustre: 443733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12004.044375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12004.050584] Lustre: Skipped 1 previous similar message [12004.135779] Lustre: lustre-MDT0001: new disk, initializing [12004.213625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12004.260818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12004.269704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12008.250678] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12016.391369] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12021.680971] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12028.341791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12028.666665] Lustre: lustre-OST0000: new disk, initializing [12028.670513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12028.682845] Lustre: 445701:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12028.769933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12034.533468] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12037.203648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12037.228687] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12037.341513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12045.202699] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12052.966681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12052.971463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12052.983041] Lustre: Skipped 2 previous similar messages [12052.998048] Lustre: Skipped 2 previous similar messages [12058.084085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12058.091439] Lustre: Skipped 2 previous similar messages [12059.254161] Lustre: server umount lustre-MDT0000 complete [12063.049822] LustreError: 444527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824893 with bad export cookie 16399336771413213725 [12063.052563] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12063.059808] LustreError: 444527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12063.203372] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12063.311660] Lustre: server umount lustre-MDT0001 complete [12069.708034] Lustre: server umount lustre-OST0000 complete [12080.209751] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 16:15:09 (1786824909) [12088.173565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12088.595431] LustreError: 447731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12088.620141] LustreError: 447731:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12088.666138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12092.941521] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12093.924516] LustreError: 447732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12099.043580] LustreError: 447731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12101.278150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12101.886815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12107.670676] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12116.428897] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12122.449720] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12142.465979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12142.650773] Lustre: Found index 0 for lustre-OST0000, updating log [12142.792079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12148.819476] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12156.098118] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12159.223988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12167.657385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12167.670985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12167.688096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12173.519524] Lustre: server umount lustre-OST0000 complete [12194.273088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12194.281850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12194.299681] Lustre: Skipped 1 previous similar message [12194.308513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12194.316135] Lustre: Skipped 3 previous similar messages [12196.788040] Lustre: server umount lustre-MDT0000 complete [12197.350749] LustreError: 448459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12197.378515] LustreError: 448459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12200.165729] LustreError: 447712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825031 with bad export cookie 16399336771413215118 [12200.187180] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12200.464996] Lustre: server umount lustre-MDT0001 complete [12210.211063] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 16:17:19 (1786825039) [12212.387192] LNet: 1 local NIs in recovery (showing 1): 192.168.206.127@tcp [12831.342154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12831.753616] LustreError: 454139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12831.814544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12835.293862] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12836.835729] LustreError: 454140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12841.963393] LustreError: 454139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12843.042187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12843.278057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12846.220400] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12852.757157] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12858.242869] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12877.547933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12878.033508] Lustre: Found index 0 for lustre-OST0000, updating log [12878.498066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12879.549817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [12886.192212] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12889.317798] LNet: 1 local NIs in recovery (showing 1): 192.168.206.127@tcp [12896.160743] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12898.791148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12898.807413] Lustre: Skipped 1 previous similar message [12898.815497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12898.831652] Lustre: Skipped 4 previous similar messages [12912.096123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12912.315867] Lustre: server umount lustre-OST0000 complete [12916.650678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12916.659340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12916.662012] Lustre: Skipped 1 previous similar message [12916.675833] Lustre: Skipped 1 previous similar message [12917.666067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12917.676056] Lustre: Skipped 1 previous similar message [12919.713997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12919.724596] Lustre: Skipped 3 previous similar messages [12921.948085] Lustre: server umount lustre-MDT0000 complete [12922.786304] LustreError: 454140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12922.798536] LustreError: 454140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12925.184556] LustreError: 454121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825756 with bad export cookie 16399336771413216203 [12925.192311] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12925.194816] LustreError: 454121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12925.375187] Lustre: server umount lustre-MDT0001 complete [12955.024878] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [12963.508409] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [13002.393448] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:30:30 (1786825830) [13022.679475] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [13025.689670] Key type lgssc unregistered [13025.997590] LNet: 460757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13026.008956] LNetError: 460757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13141.740158] LNet: Removed LNI 192.168.206.127@tcp [13142.770477] Key type .llcrypt unregistered [13142.774491] Key type ._llcrypt unregistered [13159.005962] Key type ._llcrypt registered [13159.016591] Key type .llcrypt registered [13159.155517] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [13159.918082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13159.991276] alg: No test for adler32 (adler32-zlib) [13161.217336] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [13161.445581] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [13163.128163] Key type lgssc registered [13164.427454] Lustre: Echo OBD driver; http://www.lustre.org/ [13168.587228] LNet: Added LNI 192.168.206.127@tcp100 [8/256/0/180] [13169.726580] LNet: Added LNI 192.168.206.127@tcp101 [8/256/0/180] [13170.585624] LNet: Added LNI 192.168.206.127@tcp102 [8/256/0/180] [13171.547700] LNet: Added LNI 192.168.206.127@tcp103 [8/256/0/180] [13172.314073] LNet: Added LNI 192.168.206.127@tcp104 [8/256/0/180] [13173.162662] LNet: Added LNI 192.168.206.127@tcp105 [8/256/0/180] [13174.183610] LNet: Added LNI 192.168.206.127@tcp106 [8/256/0/180] [13175.024825] LNet: Added LNI 192.168.206.127@tcp107 [8/256/0/180] [13175.916791] LNet: Added LNI 192.168.206.127@tcp108 [8/256/0/180] [13176.907970] LNet: Added LNI 192.168.206.127@tcp109 [8/256/0/180] [13177.835399] LNet: Added LNI 192.168.206.127@tcp110 [8/256/0/180] [13178.652683] LNet: Added LNI 192.168.206.127@tcp111 [8/256/0/180] [13179.587274] LNet: Added LNI 192.168.206.127@tcp112 [8/256/0/180] [13180.463938] LNet: Added LNI 192.168.206.127@tcp113 [8/256/0/180] [13181.257551] LNet: Added LNI 192.168.206.127@tcp114 [8/256/0/180] [13182.140901] LNet: Added LNI 192.168.206.127@tcp115 [8/256/0/180] [13182.853530] LNet: Added LNI 192.168.206.127@tcp116 [8/256/0/180] [13183.580276] LNet: Added LNI 192.168.206.127@tcp117 [8/256/0/180] [13184.488737] LNet: Added LNI 192.168.206.127@tcp118 [8/256/0/180] [13185.565941] LNet: Added LNI 192.168.206.127@tcp119 [8/256/0/180] [13186.481503] LNet: Added LNI 192.168.206.127@tcp120 [8/256/0/180] [13187.421392] LNet: Added LNI 192.168.206.127@tcp121 [8/256/0/180] [13188.444304] LNet: Added LNI 192.168.206.127@tcp122 [8/256/0/180] [13189.411325] LNet: Added LNI 192.168.206.127@tcp123 [8/256/0/180] [13190.326450] LNet: Added LNI 192.168.206.127@tcp124 [8/256/0/180] [13191.226726] LNet: Added LNI 192.168.206.127@tcp125 [8/256/0/180] [13192.199418] LNet: Added LNI 192.168.206.127@tcp126 [8/256/0/180] [13193.181263] LNet: Added LNI 192.168.206.127@tcp127 [8/256/0/180] [13194.186571] LNet: Added LNI 192.168.206.127@tcp128 [8/256/0/180] [13195.141052] LNet: Added LNI 192.168.206.127@tcp129 [8/256/0/180] [13196.022172] LNet: Added LNI 192.168.206.127@tcp130 [8/256/0/180] [13197.040172] LNet: Added LNI 192.168.206.127@tcp131 [8/256/0/180] [13197.807986] LNet: Added LNI 192.168.206.127@tcp132 [8/256/0/180] [13198.744822] LNet: Added LNI 192.168.206.127@tcp133 [8/256/0/180] [13199.827433] LNet: Added LNI 192.168.206.127@tcp134 [8/256/0/180] [13200.822788] LNet: Added LNI 192.168.206.127@tcp135 [8/256/0/180] [13208.055480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13208.096172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13209.202744] Lustre: 463789:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.127@tcp: can't add NID 192.168.206.127@tcp130: rc = -75 [13209.279652] LNetError: 461250:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.127 [13209.292263] LNetError: Protocol error connecting to 192.168.206.127@tcp122 at host 192.168.206.127:988: is it running a compatible version of Lustre? [13209.301767] LNetError: Skipped 4 previous similar messages [13209.422660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13209.455344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13209.567042] Lustre: lustre-MDT0000: new disk, initializing [13209.570455] LNet: 6 local NIs in recovery (showing 6): 192.168.206.127@tcp125, 192.168.206.127@tcp126, 192.168.206.127@tcp128, 192.168.206.127@tcp129, 192.168.206.127@tcp122, 192.168.206.127@tcp127 [13209.657432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13209.674697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13210.339362] LNetError: 463815:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.127 [13210.359836] LNetError: 463815:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [13210.381163] LNetError: Protocol error connecting to 192.168.206.127@tcp111 at host 192.168.206.127:988: is it running a compatible version of Lustre? [13210.406848] LNetError: Skipped 3 previous similar messages [13212.641736] LNetError: 461248:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.127 [13212.661372] LNetError: 461248:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [13212.669520] LNetError: Protocol error connecting to 192.168.206.127@tcp125 at host 192.168.206.127:988: is it running a compatible version of Lustre? [13212.680286] LNetError: Skipped 6 previous similar messages [13213.725749] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [13216.738352] LNetError: 461249:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.127 [13216.757486] LNetError: Protocol error connecting to 192.168.206.127@tcp125 at host 192.168.206.127:988: is it running a compatible version of Lustre? [13223.723922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13223.954602] Lustre: 463807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13224.082932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13224.087323] Lustre: Skipped 1 previous similar message [13224.225758] Lustre: lustre-MDT0001: new disk, initializing [13224.264984] Lustre: 464609:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.127@tcp: can't add NID 192.168.206.127@tcp130: rc = -75 [13224.272632] Lustre: 464609:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13224.390902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13224.431252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13224.443615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13224.932795] LNetError: 463817:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.127 [13224.955794] LNetError: Protocol error connecting to 192.168.206.127@tcp125 at host 192.168.206.127:988: is it running a compatible version of Lustre? [13228.451554] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [13231.149329] Lustre: 464973:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.127@tcp: can't add NID 192.168.206.127@tcp130: rc = -75 [13231.157093] Lustre: 464973:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13237.676198] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13240.294266] LNetError: 463821:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.127 [13240.317100] LNetError: Protocol error connecting to 192.168.206.127@tcp121 at host 192.168.206.127:988: is it running a compatible version of Lustre? [13243.128153] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13249.657555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13249.950398] Lustre: lustre-OST0000: new disk, initializing [13249.953792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13249.959826] Lustre: 465782:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13250.035563] Lustre: 465787:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.127@tcp: can't add NID 192.168.206.127@tcp130: rc = -75 [13250.040904] Lustre: 465787:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13250.085864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13256.103711] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [13258.300230] Lustre: 466240:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.127@tcp: can't add NID 192.168.206.127@tcp130: rc = -75 [13258.313699] Lustre: 466240:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13258.392487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13258.404618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13258.505059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13263.842443] LNetError: 463820:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.127 [13263.858433] LNetError: 463820:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [13263.866062] LNetError: Protocol error connecting to 192.168.206.127@tcp123 at host 192.168.206.127:988: is it running a compatible version of Lustre? [13263.876267] LNetError: Skipped 9 previous similar messages [13265.095196] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13274.082852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13274.106910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13274.108491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13274.125157] Lustre: Skipped 1 previous similar message [13279.211943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13279.223252] Lustre: Skipped 1 previous similar message [13284.330696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13284.342145] Lustre: Skipped 1 previous similar message [13286.880103] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13287.158362] Lustre: server umount lustre-OST0000 complete [13290.977510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13290.984721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13290.994884] Lustre: Skipped 1 previous similar message [13291.000910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13294.564114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13297.271112] Lustre: server umount lustre-MDT0000 complete [13300.647620] LustreError: 463798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826131 with bad export cookie 985736370060384052 [13300.650711] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13300.657142] LustreError: 463798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13300.915600] Lustre: server umount lustre-MDT0001 complete [13304.804909] LNetError: 463815:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.127 [13304.828506] LNetError: 463815:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [13304.842990] LNetError: Protocol error connecting to 192.168.206.127@tcp121 at host 192.168.206.127:988: is it running a compatible version of Lustre? [13304.879519] LNetError: Skipped 13 previous similar messages [13313.794281] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [13317.200211] Key type lgssc unregistered [13317.813840] LNet: 467700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13317.827204] LNetError: 467700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13317.837683] LNet: Removed LNI 192.168.206.127@tcp [13318.881561] LNet: Removed LNI 192.168.206.127@tcp100 [13318.886856] LNetError: 467700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13318.910294] LNetError: 467700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13318.933075] LNet: Removed LNI 192.168.206.127@tcp101 [13319.968207] LNet: Removed LNI 192.168.206.127@tcp102 [13319.976620] LNetError: 467700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13319.993376] LNetError: 467700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13320.008625] LNet: Removed LNI 192.168.206.127@tcp103 [13321.056211] LNet: Removed LNI 192.168.206.127@tcp104 [13322.081040] LNet: Removed LNI 192.168.206.127@tcp105 [13322.083314] LNetError: 467700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13322.091387] LNetError: 467700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13322.096225] LNet: Removed LNI 192.168.206.127@tcp106 [13323.104221] LNet: Removed LNI 192.168.206.127@tcp107 [13324.128271] LNet: Removed LNI 192.168.206.127@tcp108 [13325.152300] LNet: Removed LNI 192.168.206.127@tcp109 [13326.176339] LNet: Removed LNI 192.168.206.127@tcp110 [13326.179433] LNetError: 467700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13326.197881] LNetError: 467700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13326.207765] LNet: Removed LNI 192.168.206.127@tcp111 [13327.266626] LNet: Removed LNI 192.168.206.127@tcp112 [13328.288226] LNet: Removed LNI 192.168.206.127@tcp113 [13329.312251] LNet: Removed LNI 192.168.206.127@tcp114 [13330.336516] LNet: Removed LNI 192.168.206.127@tcp115 [13331.361056] LNet: Removed LNI 192.168.206.127@tcp116 [13332.384298] LNet: Removed LNI 192.168.206.127@tcp117 [13333.409189] LNet: Removed LNI 192.168.206.127@tcp118 [13334.432481] LNet: Removed LNI 192.168.206.127@tcp119 [13334.438847] LNetError: 467700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13334.445534] LNetError: 467700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13334.460098] LNet: Removed LNI 192.168.206.127@tcp120 [13335.520216] LNet: Removed LNI 192.168.206.127@tcp121 [13336.544390] LNet: Removed LNI 192.168.206.127@tcp122 [13337.568312] LNet: Removed LNI 192.168.206.127@tcp123 [13338.592149] LNet: Removed LNI 192.168.206.127@tcp124 [13339.616335] LNet: Removed LNI 192.168.206.127@tcp125 [13340.640157] LNet: Removed LNI 192.168.206.127@tcp126 [13341.664151] LNet: Removed LNI 192.168.206.127@tcp127 [13342.690105] LNet: Removed LNI 192.168.206.127@tcp128 [13343.714212] LNet: Removed LNI 192.168.206.127@tcp129 [13343.722219] LNet: Removed LNI 192.168.206.127@tcp130 [13343.727185] LNet: Removed LNI 192.168.206.127@tcp131 [13343.735769] LNet: Removed LNI 192.168.206.127@tcp132 [13343.744413] LNet: Removed LNI 192.168.206.127@tcp133 [13343.753471] LNet: Removed LNI 192.168.206.127@tcp134 [13343.772446] LNet: Removed LNI 192.168.206.127@tcp135 [13344.645543] Key type .llcrypt unregistered [13344.650435] Key type ._llcrypt unregistered [13367.574206] Key type ._llcrypt registered [13367.576788] Key type .llcrypt registered [13367.713754] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [13383.455147] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [13384.683541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13384.723876] alg: No test for adler32 (adler32-zlib) [13385.936964] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [13386.364460] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [13388.056438] Key type lgssc registered [13389.097642] Lustre: Echo OBD driver; http://www.lustre.org/ [13424.399079] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:37:33 (1786826253) [13441.356408] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [13443.426521] Key type lgssc unregistered [13443.662698] LNet: 471790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13443.669566] LNetError: 471790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13443.698607] LNet: Removed LNI 192.168.206.127@tcp [13444.439898] Key type .llcrypt unregistered [13444.442388] Key type ._llcrypt unregistered [13459.451668] Key type ._llcrypt registered [13459.454420] Key type .llcrypt registered [13459.612584] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [13460.428194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13460.566956] alg: No test for adler32 (adler32-zlib) [13461.650531] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [13461.839918] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [13463.496149] Key type lgssc registered [13464.485756] Lustre: Echo OBD driver; http://www.lustre.org/ [13473.644874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13473.697813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13474.936541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13474.984664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13475.109969] Lustre: lustre-MDT0000: new disk, initializing [13475.282708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13475.352939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13479.285332] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [13488.989508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13489.129972] Lustre: 473005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13489.198864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13489.202616] Lustre: Skipped 1 previous similar message [13489.265489] Lustre: lustre-MDT0001: new disk, initializing [13489.353750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13489.383727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13489.394488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13494.001143] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [13503.318350] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13508.285737] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13514.158437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13514.364367] Lustre: lustre-OST0000: new disk, initializing [13514.367128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13514.371728] Lustre: 474973:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13514.451499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13519.954968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13519.968622] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13520.108725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13520.651711] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [13529.549850] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13533.282290] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing lnet_if_list [13538.727789] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13538.757746] LNet: Added LNI 192.168.206.127@tcp10 [8/256/0/180] [13544.804804] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13544.830886] LNet: Added LNI 192.168.206.127@tcp11 [8/256/0/180] [13549.369830] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13549.392747] LNet: Added LNI 192.168.206.127@tcp12 [8/256/0/180] [13553.896292] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13553.931372] LNet: Added LNI 192.168.206.127@tcp13 [8/256/0/180] [13557.443292] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13557.484973] LNet: Added LNI 192.168.206.127@tcp14 [8/256/0/180] [13561.985380] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13562.048952] LNet: Added LNI 192.168.206.127@tcp15 [8/256/0/180] [13566.269107] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13566.303942] LNet: Added LNI 192.168.206.127@tcp16 [8/256/0/180] [13571.412739] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13571.431620] LNet: Added LNI 192.168.206.127@tcp17 [8/256/0/180] [13575.623704] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13575.677915] LNet: Added LNI 192.168.206.127@tcp18 [8/256/0/180] [13581.043313] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13581.066066] LNet: Added LNI 192.168.206.127@tcp19 [8/256/0/180] [13586.132747] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13586.159782] LNet: Added LNI 192.168.206.127@tcp20 [8/256/0/180] [13590.860238] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13590.892933] LNet: Added LNI 192.168.206.127@tcp21 [8/256/0/180] [13596.088835] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13596.110794] LNet: Added LNI 192.168.206.127@tcp22 [8/256/0/180] [13600.863583] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13600.891539] LNet: Added LNI 192.168.206.127@tcp23 [8/256/0/180] [13607.063591] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13607.120468] LNet: Added LNI 192.168.206.127@tcp24 [8/256/0/180] [13613.573959] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13613.598586] LNet: Added LNI 192.168.206.127@tcp25 [8/256/0/180] [13620.861211] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13620.893372] LNet: Added LNI 192.168.206.127@tcp26 [8/256/0/180] [13627.028121] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13627.060322] LNet: Added LNI 192.168.206.127@tcp27 [8/256/0/180] [13634.774461] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13634.799409] LNet: Added LNI 192.168.206.127@tcp28 [8/256/0/180] [13641.589834] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13641.628830] LNet: Added LNI 192.168.206.127@tcp29 [8/256/0/180] [13648.230884] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13648.257627] LNet: Added LNI 192.168.206.127@tcp30 [8/256/0/180] [13655.007282] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13655.051740] LNet: Added LNI 192.168.206.127@tcp31 [8/256/0/180] [13662.053087] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13662.080715] LNet: Added LNI 192.168.206.127@tcp32 [8/256/0/180] [13667.910434] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13667.959618] LNet: Added LNI 192.168.206.127@tcp33 [8/256/0/180] [13673.724204] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13673.766649] LNet: Added LNI 192.168.206.127@tcp34 [8/256/0/180] [13680.830444] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13680.863218] LNet: Added LNI 192.168.206.127@tcp35 [8/256/0/180] [13689.031988] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13689.083171] LNet: Added LNI 192.168.206.127@tcp36 [8/256/0/180] [13696.890343] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13696.916528] LNet: Added LNI 192.168.206.127@tcp37 [8/256/0/180] [13703.550728] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13703.597732] LNet: Added LNI 192.168.206.127@tcp38 [8/256/0/180] [13709.927819] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13709.955766] LNet: Added LNI 192.168.206.127@tcp39 [8/256/0/180] [13716.914995] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13716.939067] LNet: Added LNI 192.168.206.127@tcp40 [8/256/0/180] [13723.001584] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13723.030331] LNet: Added LNI 192.168.206.127@tcp41 [8/256/0/180] [13729.601800] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13729.641389] LNet: Added LNI 192.168.206.127@tcp42 [8/256/0/180] [13736.367475] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13736.391202] LNet: Added LNI 192.168.206.127@tcp43 [8/256/0/180] [13742.808950] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13742.836509] LNet: Added LNI 192.168.206.127@tcp44 [8/256/0/180] [13749.233899] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13749.274963] LNet: Added LNI 192.168.206.127@tcp45 [8/256/0/180] [13756.625206] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13756.651037] LNet: Added LNI 192.168.206.127@tcp46 [8/256/0/180] [13763.131878] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13763.166362] LNet: Added LNI 192.168.206.127@tcp47 [8/256/0/180] [13770.469352] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13770.489840] LNet: Added LNI 192.168.206.127@tcp48 [8/256/0/180] [13778.201189] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13778.220437] LNet: Added LNI 192.168.206.127@tcp49 [8/256/0/180] [13787.028808] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13787.055506] LNet: Added LNI 192.168.206.127@tcp50 [8/256/0/180] [13807.075327] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13807.076879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13807.085395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13812.197337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13812.209136] Lustre: Skipped 4 previous similar messages [13817.316130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13817.328316] Lustre: Skipped 2 previous similar messages [13818.336308] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13818.530107] Lustre: server umount lustre-MDT0000 complete [13823.212576] LustreError: 472997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826654 with bad export cookie 12220463620487071123 [13823.227869] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13823.236171] LustreError: 472997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13823.515393] Lustre: server umount lustre-MDT0001 complete [13843.937720] Lustre: 472278:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786826658/real 1786826658] req@00000000ec35c1ae x1873623166122368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786826674 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13843.959251] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13843.968356] Lustre: Skipped 2 previous similar messages [13844.960182] Lustre: 472280:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786826659/real 1786826659] req@00000000005d5b29 x1873623166122496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786826675 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13846.435942] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [13849.120679] Lustre: 472280:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786826663/real 1786826663] req@000000007a364227 x1873623166122624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786826679 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13854.243949] Lustre: 472277:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786826669/real 1786826669] req@000000001663d306 x1873623166123008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786826685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13854.264917] Lustre: 472277:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [13855.449050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13859.296132] Lustre: 472279:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786826674/real 1786826674] req@00000000c9296f77 x1873623166123136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786826690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13868.955652] LustreError: 474967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13868.992485] LustreError: 474967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13869.058839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13873.067462] LustreError: 474968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13873.102053] LustreError: 474968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13874.164984] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [13877.172186] LustreError: 474967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13877.175963] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13882.339594] LustreError: 474967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13882.808147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13883.173093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13886.299757] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [13888.305746] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13893.190668] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13898.421587] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13919.211094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13919.217796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13919.230261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13919.235716] Lustre: Skipped 1 previous similar message [13924.327890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13924.340746] Lustre: Skipped 1 previous similar message [13932.000222] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13932.165338] Lustre: server umount lustre-OST0000 complete [13939.684995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13939.690051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13939.691774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13939.691783] Lustre: Skipped 2 previous similar messages [13939.694650] Lustre: Skipped 2 previous similar messages [13941.990073] Lustre: server umount lustre-MDT0000 complete [13944.806976] LustreError: 499215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13944.819208] LustreError: 499215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13945.398588] LustreError: 472997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826776 with bad export cookie 12220463620487073692 [13945.399394] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13945.417084] LustreError: 472997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13945.673620] Lustre: server umount lustre-MDT0001 complete [13958.136509] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [13960.735271] Key type lgssc unregistered [13961.059781] LNet: 502019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13961.068444] LNetError: 502019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13961.079944] LNet: Removed LNI 192.168.206.127@tcp [13961.084891] LNet: Removed LNI 192.168.206.127@tcp10 [13961.088479] LNet: Removed LNI 192.168.206.127@tcp11 [13961.091988] LNet: Removed LNI 192.168.206.127@tcp12 [13961.103138] LNet: Removed LNI 192.168.206.127@tcp13 [13961.107401] LNet: Removed LNI 192.168.206.127@tcp14 [13961.109325] LNet: Removed LNI 192.168.206.127@tcp15 [13961.111874] LNet: Removed LNI 192.168.206.127@tcp16 [13961.114517] LNet: Removed LNI 192.168.206.127@tcp17 [13961.116820] LNet: Removed LNI 192.168.206.127@tcp18 [13961.120465] LNet: Removed LNI 192.168.206.127@tcp19 [13961.124250] LNet: Removed LNI 192.168.206.127@tcp20 [13961.126978] LNet: Removed LNI 192.168.206.127@tcp21 [13961.132912] LNet: Removed LNI 192.168.206.127@tcp22 [13961.136762] LNet: Removed LNI 192.168.206.127@tcp23 [13961.139226] LNet: Removed LNI 192.168.206.127@tcp24 [13961.142686] LNet: Removed LNI 192.168.206.127@tcp25 [13961.145468] LNet: Removed LNI 192.168.206.127@tcp26 [13961.148769] LNet: Removed LNI 192.168.206.127@tcp27 [13961.151957] LNet: Removed LNI 192.168.206.127@tcp28 [13961.155683] LNet: Removed LNI 192.168.206.127@tcp29 [13961.158614] LNet: Removed LNI 192.168.206.127@tcp30 [13961.161950] LNet: Removed LNI 192.168.206.127@tcp31 [13961.165156] LNet: Removed LNI 192.168.206.127@tcp32 [13961.167545] LNet: Removed LNI 192.168.206.127@tcp33 [13961.170351] LNet: Removed LNI 192.168.206.127@tcp34 [13961.173518] LNet: Removed LNI 192.168.206.127@tcp35 [13961.176392] LNet: Removed LNI 192.168.206.127@tcp36 [13961.178729] LNet: Removed LNI 192.168.206.127@tcp37 [13961.181290] LNet: Removed LNI 192.168.206.127@tcp38 [13961.183449] LNet: Removed LNI 192.168.206.127@tcp39 [13961.186192] LNet: Removed LNI 192.168.206.127@tcp40 [13961.188751] LNet: Removed LNI 192.168.206.127@tcp41 [13961.192272] LNet: Removed LNI 192.168.206.127@tcp42 [13961.194635] LNet: Removed LNI 192.168.206.127@tcp43 [13961.198124] LNet: Removed LNI 192.168.206.127@tcp44 [13961.200544] LNet: Removed LNI 192.168.206.127@tcp45 [13961.203644] LNet: Removed LNI 192.168.206.127@tcp46 [13961.205437] LNet: Removed LNI 192.168.206.127@tcp47 [13961.208109] LNet: Removed LNI 192.168.206.127@tcp48 [13961.210304] LNet: Removed LNI 192.168.206.127@tcp49 [13961.217811] LNet: Removed LNI 192.168.206.127@tcp50 [13961.795099] Key type .llcrypt unregistered [13961.796855] Key type ._llcrypt unregistered [13983.752554] Key type ._llcrypt registered [13983.756564] Key type .llcrypt registered [13983.922451] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [13999.340831] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [13999.907561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13999.936376] alg: No test for adler32 (adler32-zlib) [14000.952362] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [14001.185914] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [14002.817500] Key type lgssc registered [14003.810265] Lustre: Echo OBD driver; http://www.lustre.org/ [14036.479854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14036.523167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14037.781691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14037.805642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14037.948790] Lustre: lustre-MDT0000: new disk, initializing [14038.070502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14038.093977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14041.602132] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [14051.726722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14051.902137] Lustre: 505213:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14052.012736] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14052.017231] Lustre: Skipped 1 previous similar message [14052.116775] Lustre: lustre-MDT0001: new disk, initializing [14052.264903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14052.331523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14052.356243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14057.854851] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [14067.211730] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14073.466328] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14080.455800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14080.655418] Lustre: *** cfs_fail_loc=90f, val=0*** [14080.724211] Lustre: lustre-OST0000: new disk, initializing [14080.732165] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14080.738560] Lustre: 507230:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14080.819376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14086.209694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14086.217315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14086.362575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14086.534498] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [14095.019676] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14107.116515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14107.127859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14107.137763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14108.331444] Lustre: server umount lustre-OST0000 complete [14113.763633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14113.767897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14113.782388] Lustre: Skipped 1 previous similar message [14113.790357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14113.793821] Lustre: Skipped 1 previous similar message [14117.347797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14117.351713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14118.874674] Lustre: server umount lustre-MDT0000 complete [14122.165680] LustreError: 506009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826953 with bad export cookie 18439129762908317405 [14122.175924] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14122.183863] LustreError: 506009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14122.474791] Lustre: server umount lustre-MDT0001 complete [14132.581677] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [14134.693574] Key type lgssc unregistered [14134.944606] LNet: 509139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14134.948897] LNetError: 509139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14134.958321] LNet: Removed LNI 192.168.206.127@tcp [14135.666840] Key type .llcrypt unregistered [14135.669824] Key type ._llcrypt unregistered [14152.702942] Key type ._llcrypt registered [14152.707425] Key type .llcrypt registered [14152.821789] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [14153.847449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14153.881897] alg: No test for adler32 (adler32-zlib) [14155.073212] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [14155.323412] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [14156.968160] Key type lgssc registered [14158.051985] Lustre: Echo OBD driver; http://www.lustre.org/ [14164.491867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14164.509880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14165.847987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14169.674726] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [14175.202668] LustreError: 510411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14175.222130] LustreError: 510411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14177.158242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14177.426513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14180.783416] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [14187.656282] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14192.876564] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14199.086746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14199.366800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14204.455504] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [14212.000804] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14223.260711] loop0: detected capacity change from 0 to 204800000 [14223.391674] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14223.410291] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14223.433122] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14230.928544] loop0: detected capacity change from 0 to 204800000 [14231.077759] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14231.093942] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14231.116950] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14235.783783] loop0: detected capacity change from 0 to 204800000 [14241.588908] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14241.690843] Lustre: fs15246-OST0000: new disk, initializing [14241.693534] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14241.698215] Lustre: 514005:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14241.741028] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14246.884627] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [14257.449713] loop1: detected capacity change from 0 to 204800000 [14262.946871] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14263.005918] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14263.040457] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14263.095758] Lustre: fs15246-MDT0000: new disk, initializing [14263.192983] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14263.216657] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14263.370091] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14264.419662] Lustre: 515127:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000cbc14a12 x1873623893718144/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786827111 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14267.552558] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [14278.641490] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14278.659674] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14278.715902] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14288.870567] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14288.884305] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14293.991902] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14295.026804] Lustre: server umount fs15246-MDT0000 complete [14299.109072] LustreError: 510408:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14304.007620] Lustre: server umount fs15246-OST0000 complete [14314.473247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14314.479874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14314.488682] Lustre: Skipped 1 previous similar message [14314.500808] Lustre: Skipped 1 previous similar message [14319.589059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14319.594914] Lustre: Skipped 1 previous similar message [14324.706192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14324.710557] Lustre: Skipped 1 previous similar message [14328.288340] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14328.435844] Lustre: server umount lustre-OST0000 complete [14334.948203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14334.963959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14337.931829] Lustre: server umount lustre-MDT0000 complete [14340.065842] LustreError: 511139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14340.096834] LustreError: 511139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14341.493664] LustreError: 510392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827172 with bad export cookie 4050254368274744229 [14341.494729] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14341.502729] LustreError: 510392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14342.056351] Lustre: server umount lustre-MDT0001 complete [14351.579611] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [14353.805510] Key type lgssc unregistered [14354.038819] LNet: 518225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14354.047030] LNetError: 518225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14355.113571] LNet: Removed LNI 192.168.206.127@tcp [14355.828222] Key type .llcrypt unregistered [14355.830653] Key type ._llcrypt unregistered [14373.520831] Key type ._llcrypt registered [14373.522854] Key type .llcrypt registered [14373.628956] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [14374.689844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14374.737665] alg: No test for adler32 (adler32-zlib) [14375.780784] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [14376.026187] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [14377.648132] Key type lgssc registered [14378.696070] Lustre: Echo OBD driver; http://www.lustre.org/ [14416.686825] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [14425.078670] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [14455.729139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14455.750453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14456.948848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14456.976634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14457.086225] Lustre: lustre-MDT0000: new disk, initializing [14457.196703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14457.213412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14461.653312] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [14470.701816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14470.812768] Lustre: 522693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14470.857660] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14470.861902] Lustre: Skipped 1 previous similar message [14470.919291] Lustre: lustre-MDT0001: new disk, initializing [14470.998966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14471.028166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14471.043230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14475.435851] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [14486.547262] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14491.777607] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14498.413886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14498.631293] Lustre: lustre-OST0000: new disk, initializing [14498.637785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14498.642720] Lustre: 524661:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14498.729710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14504.607576] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [14508.085345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14508.093050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14508.253454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14513.506693] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14517.784197] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14517.995811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14522.613306] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14522.788343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14528.484290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14528.491701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14528.507493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14528.993672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14529.010666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14529.028717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14530.881440] Lustre: server umount lustre-OST0000 complete [14534.117760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14534.143383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14534.151057] Lustre: Skipped 1 previous similar message [14539.233678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14539.240708] Lustre: Skipped 1 previous similar message [14540.421463] Lustre: server umount lustre-MDT0000 complete [14543.621482] LustreError: 522685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827374 with bad export cookie 14566276097963435611 [14543.631444] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14543.631618] LustreError: 522685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14543.929664] Lustre: server umount lustre-MDT0001 complete [14568.521356] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [14580.100155] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [14613.051499] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14341 sec ========= 16:57:22 (1786827442) [14614.681122] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:57:23 (1786827443) === [14618.113394] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:57:27 (1786827447) === [14655.802115] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [14658.579499] Key type lgssc unregistered [14658.758950] LNet: 531310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14658.764413] LNetError: 531310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14659.814650] LNet: Removed LNI 192.168.206.127@tcp [14660.681895] Key type .llcrypt unregistered [14660.684384] Key type ._llcrypt unregistered