[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 498724027 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002378] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004019] kvm-guest: setup PV IPIs [ 0.007672] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.011110] Yama: becoming mindful. [ 0.012053] SELinux: Initializing. [ 0.013072] *** VALIDATE selinux *** [ 0.022413] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027175] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030108] *** VALIDATE tmpfs *** [ 0.031410] *** VALIDATE proc *** [ 0.032232] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035260] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036140] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038035] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042225] debug: unmapping init [mem 0xffffffffa7e59000-0xffffffffa7e60fff] [ 0.045186] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046676] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047032] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049019] ... generic registers: 4 [ 0.050016] ... value mask: 0000ffffffffffff [ 0.051018] ... max period: 00007fffffffffff [ 0.052017] ... fixed-purpose events: 3 [ 0.053016] ... event mask: 000000070000000f [ 0.055230] rcu: Hierarchical SRCU implementation. [ 0.057393] smp: Bringing up secondary CPUs ... [ 0.058656] x86: Booting SMP configuration: [ 0.059030] .... node #0, CPUs: #1 #2 #3 [ 0.063224] smp: Brought up 1 node, 4 CPUs [ 0.065022] smpboot: Max logical packages: 1 [ 0.066018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.226027] node 0 deferred pages initialised in 158ms [ 0.230195] devtmpfs: initialized [ 0.231327] x86/mm: Memory block size: 128MB [ 0.235000] gcov: version magic: 0x41383552 [ 0.237391] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.238095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.240423] pinctrl core: initialized pinctrl subsystem [ 0.242269] [ 0.242793] ************************************************************* [ 0.245013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.247013] ** ** [ 0.249019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.252020] ** ** [ 0.254016] ** This means that this kernel is built to expose internal ** [ 0.256015] ** IOMMU data structures, which may compromise security on ** [ 0.258013] ** your system. ** [ 0.259013] ** ** [ 0.262018] ** If you see this message and you are not debugging the ** [ 0.264016] ** kernel, report this immediately to your vendor! ** [ 0.267016] ** ** [ 0.270028] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.272019] ************************************************************* [ 0.275619] NET: Registered protocol family 16 [ 0.277746] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.281064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.284067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.288423] cpuidle: using governor menu [ 0.291494] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.294575] PCI: Using configuration type 1 for base access [ 0.296130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.306150] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.309052] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.313131] cryptd: max_cpu_qlen set to 1000 [ 0.317510] ACPI: Added _OSI(Module Device) [ 0.319026] ACPI: Added _OSI(Processor Device) [ 0.321025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.323023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.329467] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.337563] ACPI: Interpreter enabled [ 0.339075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.340016] ACPI: Using IOAPIC for interrupt routing [ 0.343152] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.346280] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.357467] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.360056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.363023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.367085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.373366] acpiphp: Slot [2] registered [ 0.375286] acpiphp: Slot [5] registered [ 0.377202] acpiphp: Slot [6] registered [ 0.379173] acpiphp: Slot [7] registered [ 0.380184] acpiphp: Slot [8] registered [ 0.382162] acpiphp: Slot [9] registered [ 0.384169] acpiphp: Slot [10] registered [ 0.386172] acpiphp: Slot [3] registered [ 0.387111] acpiphp: Slot [4] registered [ 0.389085] acpiphp: Slot [11] registered [ 0.392153] acpiphp: Slot [12] registered [ 0.394112] acpiphp: Slot [13] registered [ 0.396128] acpiphp: Slot [14] registered [ 0.397141] acpiphp: Slot [15] registered [ 0.399124] acpiphp: Slot [16] registered [ 0.401118] acpiphp: Slot [17] registered [ 0.402106] acpiphp: Slot [18] registered [ 0.404138] acpiphp: Slot [19] registered [ 0.405102] acpiphp: Slot [20] registered [ 0.407098] acpiphp: Slot [21] registered [ 0.409120] acpiphp: Slot [22] registered [ 0.411118] acpiphp: Slot [23] registered [ 0.412125] acpiphp: Slot [24] registered [ 0.414130] acpiphp: Slot [25] registered [ 0.416113] acpiphp: Slot [26] registered [ 0.417122] acpiphp: Slot [27] registered [ 0.419110] acpiphp: Slot [28] registered [ 0.421128] acpiphp: Slot [29] registered [ 0.422151] acpiphp: Slot [30] registered [ 0.424118] acpiphp: Slot [31] registered [ 0.426068] PCI host bridge to bus 0000:00 [ 0.428027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.430025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.432026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.435028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.437033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.441017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.443208] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.447992] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.451373] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.463012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.468060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.471029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.474023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.477025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.480604] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.484840] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.488047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.490747] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.496018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.510016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.516028] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.523194] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.542018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.552025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.570018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.583850] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.592018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.598016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.622023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.643931] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.652019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.662020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.685025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.702614] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.711024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.720025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.739027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.750230] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.759019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.772032] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.797022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.808717] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.820016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.830022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.846025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.856333] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.859431] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.862407] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.864371] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.868256] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.874164] iommu: Default domain type: Passthrough [ 0.876512] SCSI subsystem initialized [ 0.878189] ACPI: bus type USB registered [ 0.880134] usbcore: registered new interface driver usbfs [ 0.881141] usbcore: registered new interface driver hub [ 0.883113] usbcore: registered new device driver usb [ 0.885165] pps_core: LinuxPPS API ver. 1 registered [ 0.886010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.889093] PTP clock support registered [ 0.892076] EDAC MC: Ver: 3.0.0 [ 0.894197] PCI: Using ACPI for IRQ routing [ 0.896324] NetLabel: Initializing [ 0.898020] NetLabel: domain hash size = 128 [ 0.899018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.902116] NetLabel: unlabeled traffic allowed by default [ 0.905107] vgaarb: loaded [ 0.906359] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.909019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.917426] clocksource: Switched to clocksource kvm-clock [ 1.037475] VFS: Disk quotas dquot_6.6.0 [ 1.039172] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.042297] *** VALIDATE ramfs *** [ 1.044069] *** VALIDATE hugetlbfs *** [ 1.046227] pnp: PnP ACPI init [ 1.049326] pnp: PnP ACPI: found 6 devices [ 1.068221] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.072135] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.074470] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.076957] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.079513] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.082386] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.085072] NET: Registered protocol family 2 [ 1.087766] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.092880] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.096874] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.103301] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.107093] TCP: Hash tables configured (established 65536 bind 65536) [ 1.110336] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.114158] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.117330] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.120528] NET: Registered protocol family 1 [ 1.123391] RPC: Registered named UNIX socket transport module. [ 1.126540] RPC: Registered udp transport module. [ 1.129183] RPC: Registered tcp transport module. [ 1.131681] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.134907] NET: Registered protocol family 44 [ 1.136826] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.140235] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.142996] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.145928] PCI: CLS 0 bytes, default 64 [ 1.148484] Unpacking initramfs... [ 2.755707] debug: unmapping init [mem 0xffff8aab3cc54000-0xffff8aab3ffbffff] [ 2.760381] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.762766] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.765873] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.282704] Initialise system trusted keyrings [ 3.284638] Key type blacklist registered [ 3.287077] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.296652] zbud: loaded [ 3.300240] *** VALIDATE nfs *** [ 3.301497] *** VALIDATE nfs4 *** [ 3.303192] pstore: using deflate compression [ 3.306790] Platform Keyring initialized [ 3.430395] NET: Registered protocol family 38 [ 3.432126] Key type asymmetric registered [ 3.433647] Asymmetric key parser 'x509' registered [ 3.435655] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.439077] io scheduler mq-deadline registered [ 3.440619] io scheduler kyber registered [ 3.442262] io scheduler bfq registered [ 3.444281] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.447838] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.451186] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.454585] ACPI: Power Button [PWRF] [ 3.460688] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.467549] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.484379] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.491887] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.506866] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.535742] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.566769] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.572421] Non-volatile memory driver v1.3 [ 3.574562] Linux agpgart interface v0.103 [ 3.609516] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.613336] vda: detected capacity change from 0 to 76681216 [ 3.629218] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.632640] vdb: detected capacity change from 0 to 1073741824 [ 3.649194] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.652809] vdc: detected capacity change from 0 to 2621440000 [ 3.668866] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.672588] vdd: detected capacity change from 0 to 2621440000 [ 3.688228] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.691161] vde: detected capacity change from 0 to 4294967296 [ 3.707222] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.709847] vdf: detected capacity change from 0 to 4294967296 [ 3.716186] libphy: Fixed MDIO Bus: probed [ 3.721162] usbcore: registered new interface driver usbserial_generic [ 3.723365] usbserial: USB Serial support registered for generic [ 3.725405] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.729232] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.730770] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.733121] mousedev: PS/2 mouse device common for all mice [ 3.736599] rtc_cmos 00:05: RTC can wake from S4 [ 3.740879] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.742795] rtc_cmos 00:05: registered as rtc0 [ 3.746984] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.747769] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.751179] intel_pstate: CPU model not supported [ 3.755070] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.758961] hid: raw HID events driver (C) Jiri Kosina [ 3.761538] usbcore: registered new interface driver usbhid [ 3.763930] usbhid: USB HID core driver [ 3.765943] drop_monitor: Initializing network drop monitor service [ 3.768414] Initializing XFRM netlink socket [ 3.770080] NET: Registered protocol family 10 [ 3.773454] Segment Routing with IPv6 [ 3.774711] NET: Registered protocol family 17 [ 3.776654] mpls_gso: MPLS GSO support [ 3.783074] RAS: Correctable Errors collector initialized. [ 3.784841] AVX version of gcm_enc/dec engaged. [ 3.786237] AES CTR mode by8 optimization enabled [ 3.861560] sched_clock: Marking stable (3861537802, 0)->(4811264910, -949727108) [ 3.865335] registered taskstats version 1 [ 3.867489] Loading compiled-in X.509 certificates [ 3.869609] zswap: loaded using pool lzo/zbud [ 3.891848] Key type big_key registered [ 3.904633] Key type encrypted registered [ 3.906312] ima: No TPM chip found, activating TPM-bypass! [ 3.911450] ima: Allocated hash algorithm: sha1 [ 3.913323] ima: No architecture policies found [ 3.915318] evm: Initialising EVM extended attributes: [ 3.917322] evm: security.selinux [ 3.918525] evm: security.ima [ 3.919630] evm: security.capability [ 3.921192] evm: HMAC attrs: 0x1 [ 3.926108] rtc_cmos 00:05: setting system clock to 2026-09-06 08:36:50 UTC (1788683810) [ 3.935387] debug: unmapping init [mem 0xffffffffa8e03000-0xffffffffa8ffffff] [ 3.939165] debug: unmapping init [mem 0xffffffffa7b82000-0xffffffffa7e58fff] [ 3.948095] Write protecting the kernel read-only data: 28672k [ 3.951905] debug: unmapping init [mem 0xffffffffa6203000-0xffffffffa63fffff] [ 3.954735] debug: unmapping init [mem 0xffffffffa6b14000-0xffffffffa6bfffff] [ 3.997859] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.006184] systemd[1]: Detected virtualization kvm. [ 4.007948] systemd[1]: Detected architecture x86-64. [ 4.010034] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.040090] systemd[1]: No hostname configured. [ 4.043361] systemd[1]: Set hostname to . [ 4.045742] random: systemd: uninitialized urandom read (16 bytes read) [ 4.050538] systemd[1]: Initializing machine ID from random generator. [ 4.105990] random: ln: uninitialized urandom read (6 bytes read) [ 4.200378] random: systemd: uninitialized urandom read (16 bytes read) [ 4.203729] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.208515] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.218227] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.862606] device-mapper: uevent: version 1.0.3 [ 4.865261] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.635844] virtio_net virtio0 ens2: renamed from eth0 [ 5.659003] random: fast init done [ 5.715073] scsi host0: ata_piix [ 5.728177] scsi host1: ata_piix [ 5.730892] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.733598] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.584759] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.429330] random: crng init done [ 10.430778] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.995720] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.240584] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.510257] SELinux: Disabled at runtime. [ 12.567568] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.576737] systemd[1]: Detected virtualization kvm. [ 12.579107] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.134052] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.137466] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.142165] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.146478] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.149927] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.160781] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.171496] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ 13.236392] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.682113] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.047658] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.115768] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.224157] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.245887] EDAC sbridge: Ver: 1.1.2 [ 15.939023] Key type dns_resolver registered [ 16.247200] NFS: Registering the id_resolver key type [ 16.249270] Key type id_resolver registered [ 16.251222] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg360-server login: [ 42.957826] libcfs: loading out-of-tree module taints kernel. [ 42.979933] Key type ._llcrypt registered [ 42.981846] Key type .llcrypt registered [ 43.040541] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 52.521472] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 53.336181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.344908] alg: No test for adler32 (adler32-zlib) [ 54.409370] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 54.807772] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 56.464226] Key type lgssc registered [ 57.275858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 67.145024] hrtimer: interrupt took 7534814 ns [ 78.007153] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 122.875414] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 137.250511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 137.289625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.668447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 138.703200] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 138.799087] Lustre: lustre-MDT0000: new disk, initializing [ 138.881083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 138.898506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 144.162413] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.017809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.197427] Lustre: 6485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 161.283506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 161.290572] Lustre: Skipped 1 previous similar message [ 161.460995] Lustre: lustre-MDT0001: new disk, initializing [ 161.585516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 161.659202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 161.684987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 168.127800] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.909547] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 185.518988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 185.866989] Lustre: lustre-OST0000: new disk, initializing [ 185.873506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 185.883110] Lustre: 8422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 185.980859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.723359] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.616590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 195.635773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 195.733982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 208.734910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 208.901272] Lustre: lustre-OST0001: new disk, initializing [ 208.904715] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 208.913292] Lustre: 9493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 208.976817] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.574398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 214.586042] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 214.662461] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 215.532954] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.212718] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.615568] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 246.410490] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing check_logdir /tmp/testlogs/ [ 251.382912] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing yml_node [ 256.124306] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 258.415807] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 260.686165] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 262.086525] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 6 04:41:06 EDT 2026 [ 278.480348] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 280.215294] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 296.418346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.419694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.440137] Lustre: Skipped 3 previous similar messages [ 296.452874] Lustre: Skipped 3 previous similar messages [ 301.543333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.556904] Lustre: Skipped 3 previous similar messages [ 302.492292] Lustre: server umount lustre-MDT0000 complete [ 305.685194] LustreError: 10367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684112 with bad export cookie 6186582896729429107 [ 305.686535] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 305.694583] LustreError: 10367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 306.662338] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.666518] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 306.683446] Lustre: Skipped 1 previous similar message [ 306.693875] Lustre: Skipped 1 previous similar message [ 310.689702] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.702235] Lustre: Skipped 1 previous similar message [ 312.070199] Lustre: server umount lustre-MDT0001 complete [ 315.875712] Lustre: server umount lustre-OST0000 complete [ 320.369362] Lustre: server umount lustre-OST0001 complete [ 327.533188] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 339.826832] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 374.474657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.656623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 374.685803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 374.774815] Lustre: lustre-MDT0000: new disk, initializing [ 374.847287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 374.865174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 379.435959] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 390.112203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.266750] Lustre: 15047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 390.333435] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 390.337849] Lustre: Skipped 1 previous similar message [ 390.444746] Lustre: lustre-MDT0001: new disk, initializing [ 390.580468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 390.655712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 390.677833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 395.592058] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 405.393732] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 411.276756] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 418.377300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 418.671819] Lustre: lustre-OST0000: new disk, initializing [ 418.682688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 418.688940] Lustre: 17016:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 418.806962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 420.582043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 420.599676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 420.720579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 424.910116] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 433.849396] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 438.828863] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 439.057477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 443.492262] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 443.733100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 446.447424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 446.449529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.466866] LustreError: Skipped 1 previous similar message [ 446.471299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 446.481069] Lustre: Skipped 1 previous similar message [ 446.517447] Lustre: Skipped 1 previous similar message [ 451.885912] Lustre: server umount lustre-OST0000 complete [ 456.688648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.696055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 456.711191] Lustre: Skipped 1 previous similar message [ 456.716615] Lustre: Skipped 2 previous similar messages [ 461.803028] LustreError: 15914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 461.824057] LustreError: 15914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 461.974657] Lustre: server umount lustre-MDT0000 complete [ 465.542888] LustreError: 15040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684272 with bad export cookie 6186582896729430640 [ 465.545375] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 465.553242] LustreError: 15040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 465.834811] Lustre: server umount lustre-MDT0001 complete [ 469.353128] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:44:34 (1788684274) [ 471.391246] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 473.139757] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:44:37 (1788684277) [ 479.626367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.031941] LustreError: 19425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.116412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 484.532362] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 485.345650] LustreError: 19426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.445147] LustreError: 19425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.629470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.480215] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 505.596603] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 510.702912] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 517.217477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 523.704756] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 530.485808] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 539.107524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 539.115478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.134418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 539.143452] Lustre: Skipped 2 previous similar messages [ 540.134254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 544.517392] Lustre: server umount lustre-OST0000 complete [ 550.370455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.380882] Lustre: Skipped 2 previous similar messages [ 550.384693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 553.837327] Lustre: server umount lustre-MDT0000 complete [ 555.490660] LustreError: 19426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.520199] LustreError: 19426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 557.944472] LustreError: 20161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684364 with bad export cookie 6186582896729431522 [ 557.962120] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 557.968632] LustreError: 20161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 558.365543] Lustre: server umount lustre-MDT0001 complete [ 572.392640] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 575.346319] Key type lgssc unregistered [ 575.660697] LNet: 22955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 575.670118] LNetError: 22955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 576.745409] LNet: Removed LNI 192.168.203.160@tcp [ 577.617141] Key type .llcrypt unregistered [ 577.624087] Key type ._llcrypt unregistered [ 600.243660] Key type ._llcrypt registered [ 600.246229] Key type .llcrypt registered [ 600.351342] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 601.401107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 601.475780] alg: No test for adler32 (adler32-zlib) [ 602.561241] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 602.812234] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 604.456156] Key type lgssc registered [ 605.620034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 614.075491] loop: module loaded [ 614.171796] loop0: detected capacity change from 0 to 204800000 [ 614.455363] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.464322] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.865536] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 621.013589] loop0: detected capacity change from 0 to 204800000 [ 621.170776] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 621.203484] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 621.226370] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 626.830915] loop0: detected capacity change from 0 to 204800000 [ 631.576211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 631.620210] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 632.809639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 632.926858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 633.031739] Lustre: lustre-MDT0000: new disk, initializing [ 633.128174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 633.158384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 637.342789] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 646.790451] loop1: detected capacity change from 0 to 204800000 [ 652.748346] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 653.013362] Lustre: lustre-OST07c6: new disk, initializing [ 653.018347] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 653.029981] Lustre: Skipped 1 previous similar message [ 653.035400] Lustre: 26136:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 653.117140] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 659.040330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 659.046126] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 659.081831] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 660.337681] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 664.809499] Lustre: 25123:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 664.836205] Lustre: 25123:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 664.847607] Lustre: 25123:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 664.858083] Lustre: 25123:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 664.867409] Lustre: 25123:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 664.877384] Lustre: 25123:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 686.539662] Lustre: 26426:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 686.553515] Lustre: 26426:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 686.560749] Lustre: 26426:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 686.571640] Lustre: 26426:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 686.579562] Lustre: 26426:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 686.594739] Lustre: 26426:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 691.322623] Lustre: 26426:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 691.341293] Lustre: 26426:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 691.349226] Lustre: 26426:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 691.361543] Lustre: 26426:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.370370] Lustre: 26426:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 691.384941] Lustre: 26426:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.396277] Lustre: 26426:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 691.406727] Lustre: 26426:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.415654] Lustre: 26426:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 691.423588] Lustre: 26426:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.426325] Lustre: 26426:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 691.445452] Lustre: 26426:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.666149] Lustre: 25123:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 694.684198] Lustre: 25123:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 694.692538] Lustre: 25123:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 694.700823] Lustre: 25123:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.714042] Lustre: 25123:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 694.726961] Lustre: 25123:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.734834] Lustre: 25123:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 694.739732] Lustre: 25123:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.745505] Lustre: 25123:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 694.752131] Lustre: 25123:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.764594] Lustre: 25123:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 694.769761] Lustre: 25123:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.549609] Lustre: 25124:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 699.568212] Lustre: 25124:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 699.578029] Lustre: 25124:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 699.587196] Lustre: 25124:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.598361] Lustre: 25124:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 699.610143] Lustre: 25124:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.620354] Lustre: 25124:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 699.630768] Lustre: 25124:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.635902] Lustre: 25124:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 699.642773] Lustre: 25124:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.652894] Lustre: 25124:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 699.657906] Lustre: 25124:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 710.817235] Lustre: 26426:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 710.828160] Lustre: 26426:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 710.844785] Lustre: 26426:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 710.858500] Lustre: 26426:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 710.872074] Lustre: 26426:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 710.880394] Lustre: 26426:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 710.894645] Lustre: 26426:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 710.919785] Lustre: 26426:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 710.932685] Lustre: 26426:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 710.944091] Lustre: 26426:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 710.950278] Lustre: 26426:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 710.963525] Lustre: 26426:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 761.315614] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 761.317797] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 761.343399] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 765.138992] Lustre: server umount lustre-OST07c6 complete [ 773.297086] Lustre: server umount lustre-MDT0000 complete [ 787.451227] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 790.012726] Key type lgssc unregistered [ 790.274849] LNet: 28698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.280922] LNetError: 28698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.312830] LNet: Removed LNI 192.168.203.160@tcp [ 791.062354] Key type .llcrypt unregistered [ 791.066228] Key type ._llcrypt unregistered [ 806.218809] Key type ._llcrypt registered [ 806.221247] Key type .llcrypt registered [ 806.317173] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 806.968109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 807.010089] alg: No test for adler32 (adler32-zlib) [ 808.004478] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 808.216176] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 809.880299] Key type lgssc registered [ 810.685441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 817.143284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 817.167706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.510463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 822.607657] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 828.903931] LustreError: 29943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 828.929670] LustreError: 29943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 830.731734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 831.206023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 835.571572] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 843.268835] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 849.427534] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 856.450040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 856.817606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 864.003062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 864.061498] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 872.257673] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 889.827607] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 889.837602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.871547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 891.872993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 891.876120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 891.883875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 895.129841] Lustre: server umount lustre-OST0000 complete [ 902.115400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 902.116410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 902.122082] Lustre: Skipped 1 previous similar message [ 902.127044] Lustre: Skipped 2 previous similar messages [ 904.552054] Lustre: server umount lustre-MDT0000 complete [ 907.233741] LustreError: 29940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 907.255204] LustreError: 29940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 907.441754] LustreError: 32642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684714 with bad export cookie 15464692777846860929 [ 907.444362] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 907.458643] LustreError: 32642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 907.703928] Lustre: server umount lustre-MDT0001 complete [ 919.070729] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 922.147920] Key type lgssc unregistered [ 922.436278] LNet: 33682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.459514] LNetError: 33682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.470892] LNet: Removed LNI 192.168.203.160@tcp [ 923.352653] Key type .llcrypt unregistered [ 923.355212] Key type ._llcrypt unregistered [ 942.150525] Key type ._llcrypt registered [ 942.153522] Key type .llcrypt registered [ 942.233776] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 943.109370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 943.146067] alg: No test for adler32 (adler32-zlib) [ 944.175464] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 944.440320] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 946.120582] Key type lgssc registered [ 947.233777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 955.255313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 955.291722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 956.999136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 961.372360] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 967.138072] LustreError: 34948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 967.161054] LustreError: 34948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 970.300569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 970.738361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 976.112477] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 984.302566] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 989.939871] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 996.550462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 997.043681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.174370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1002.647988] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1009.775728] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1022.947445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1022.958997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1022.978983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1026.028753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1026.035373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1026.053053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.042708] Lustre: server umount lustre-OST0000 complete [ 1036.259200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.293362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1036.296508] Lustre: Skipped 1 previous similar message [ 1039.281027] Lustre: server umount lustre-MDT0000 complete [ 1041.378684] LustreError: 34948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.419434] LustreError: 34948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1043.898643] LustreError: 34928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684850 with bad export cookie 1055253603680970661 [ 1043.927108] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1044.570954] Lustre: server umount lustre-MDT0001 complete [ 1057.100662] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1059.401608] Key type lgssc unregistered [ 1059.625062] LNet: 38501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.633633] LNetError: 38501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.655789] LNet: Removed LNI 192.168.203.160@tcp [ 1060.326812] Key type .llcrypt unregistered [ 1060.329155] Key type ._llcrypt unregistered [ 1082.405909] Key type ._llcrypt registered [ 1082.418545] Key type .llcrypt registered [ 1082.621202] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1083.789900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1083.831861] alg: No test for adler32 (adler32-zlib) [ 1084.903733] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 1085.213577] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1086.920235] Key type lgssc registered [ 1088.032145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1095.501610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1095.517372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.014844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.746928] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1107.427116] LustreError: 39914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1107.447995] LustreError: 39914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1107.967875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1108.325156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1112.307297] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1119.102834] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1124.239294] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1129.950181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1130.251494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1135.492374] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1142.462803] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1142.573651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1149.413326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1149.432043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.447621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1150.415221] Lustre: server umount lustre-MDT0000 complete [ 1152.995605] LustreError: 40641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1152.997455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.028140] Lustre: Skipped 1 previous similar message [ 1154.451607] LustreError: 39895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684961 with bad export cookie 7483987524497076797 [ 1154.477087] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1154.715490] LustreError: 41598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1154.732069] LustreError: 41598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1154.763502] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.60@tcp (stopping) [ 1154.769183] Lustre: Skipped 1 previous similar message [ 1158.117079] LustreError: 39911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.129224] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.144197] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1160.861460] Lustre: server umount lustre-MDT0001 complete [ 1182.117949] Lustre: server umount lustre-OST0000 complete [ 1198.866326] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1201.855265] Key type lgssc unregistered [ 1202.205631] LNet: 43775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.210634] LNetError: 43775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1202.221915] LNet: Removed LNI 192.168.203.160@tcp [ 1203.254547] Key type .llcrypt unregistered [ 1203.259398] Key type ._llcrypt unregistered [ 1223.805735] Key type ._llcrypt registered [ 1223.809619] Key type .llcrypt registered [ 1223.901396] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1224.683098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1224.716809] alg: No test for adler32 (adler32-zlib) [ 1225.771351] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 1225.989366] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1227.681777] Key type lgssc registered [ 1228.627772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1235.110072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1235.124561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1236.477951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.449802] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1246.691776] LustreError: 45166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.717567] LustreError: 45166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1248.261353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1248.757712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1253.435533] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1261.259375] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1267.012804] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1273.396228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1273.769669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1279.591985] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1281.978236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1286.542738] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1294.522752] Lustre: lustre-OST0000: Not available for connect from 192.168.203.60@tcp (stopping) [ 1295.339496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1295.349982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.381357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1297.380615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1297.383281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1297.387787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.159970] Lustre: server umount lustre-OST0000 complete [ 1317.863143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.865578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1317.877112] Lustre: Skipped 1 previous similar message [ 1317.891183] Lustre: Skipped 2 previous similar messages [ 1322.977706] LustreError: 45162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1322.993111] LustreError: 45162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1324.545479] Lustre: server umount lustre-MDT0000 complete [ 1327.571184] LustreError: 45148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788685134 with bad export cookie 15394519798611025182 [ 1327.577890] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1327.578264] LustreError: 45148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1333.889975] Lustre: server umount lustre-MDT0001 complete [ 1344.349449] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1347.109682] Key type lgssc unregistered [ 1347.372398] LNet: 48874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1347.382452] LNetError: 48874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1347.419968] LNet: Removed LNI 192.168.203.160@tcp [ 1348.288619] Key type .llcrypt unregistered [ 1348.292915] Key type ._llcrypt unregistered [ 1373.056531] Key type ._llcrypt registered [ 1373.058042] Key type .llcrypt registered [ 1373.193355] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1374.597134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1374.668629] alg: No test for adler32 (adler32-zlib) [ 1375.754548] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 1376.082095] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1377.802772] Key type lgssc registered [ 1380.152205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1389.454196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1389.492983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.990308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1395.724738] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1401.314491] LustreError: 50289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1401.338584] LustreError: 50289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1405.858087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.624571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1411.160969] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1420.573894] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1426.019114] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1433.021904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1433.341293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1436.406627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1439.893326] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1448.527100] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1458.228497] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1461.566168] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1473.374316] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-06 5h01m17s [ 1474.638650] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1476.796126] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.60@tcp (stopping) [ 1477.606959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1477.608212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1477.634743] Lustre: Skipped 2 previous similar messages [ 1477.653117] Lustre: Skipped 1 previous similar message [ 1478.689521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1478.702627] Lustre: Skipped 3 previous similar messages [ 1482.751686] Lustre: server umount lustre-MDT0000 complete [ 1482.910378] LustreError: 50283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.783340] LustreError: 50289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.802040] LustreError: 50289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1486.573926] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1489.059455] LustreError: 51615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.922575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1493.017807] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1493.238433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1493.346652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1494.281854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1498.627797] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1501.704924] Lustre: DEBUG MARKER: Wait for df (36689) ... [ 1503.518988] Lustre: DEBUG MARKER: done [ 1508.323106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1508.336676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1508.348338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1508.835583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1512.449235] Lustre: server umount lustre-OST0000 complete [ 1520.102968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1520.114837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1520.134047] Lustre: Skipped 2 previous similar messages [ 1520.134371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1520.137939] Lustre: Skipped 3 previous similar messages [ 1530.336112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1530.562890] Lustre: server umount lustre-MDT0000 complete [ 1533.422289] LustreError: 50289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1533.465801] LustreError: 50289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1534.983648] LustreError: 50269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788685341 with bad export cookie 5385648042121648927 [ 1534.986569] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1534.999070] LustreError: 50269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1535.475733] Lustre: server umount lustre-MDT0001 complete [ 1546.440794] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1549.202753] Key type lgssc unregistered [ 1549.527718] LNet: 55225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.533688] LNetError: 55225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.559391] LNet: Removed LNI 192.168.203.160@tcp [ 1550.466756] Key type .llcrypt unregistered [ 1550.471367] Key type ._llcrypt unregistered [ 1586.638472] Key type ._llcrypt registered [ 1586.641361] Key type .llcrypt registered [ 1586.762639] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1587.558060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1587.603085] alg: No test for adler32 (adler32-zlib) [ 1588.722963] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 1589.036290] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1590.752239] Key type lgssc registered [ 1591.670724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1599.678903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1599.696544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1600.905812] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1600.923425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1601.014473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.050371] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1615.469256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.617325] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1615.699903] Lustre: 57436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1616.166442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1621.423389] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1628.297122] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1633.543253] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1640.235884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1640.400269] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1640.586055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1646.735721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1647.685743] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1654.739916] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1661.855172] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-06 5h04m26s [ 1663.418479] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1670.741572] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1673.427182] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1673.657304] LustreError: 58446:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1693.915137] Lustre: lustre-MDT0000: Client 7fe1987e-627f-4490-bc55-aaa6967bb3d7 (at 192.168.203.60@tcp) reconnecting [ 1713.680390] LustreError: 58446:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1716.086973] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1718.603876] Lustre: DEBUG MARKER: done [ 1728.481200] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1728.494402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1728.519631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1728.998574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1729.015293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1729.025818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1734.113729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1734.118340] Lustre: Skipped 1 previous similar message [ 1739.232225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1739.234702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1739.283516] Lustre: Skipped 1 previous similar message [ 1739.541891] Lustre: server umount lustre-OST0000 complete [ 1744.356140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1744.357664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1744.383579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1744.406035] Lustre: Skipped 1 previous similar message [ 1744.417233] Lustre: Skipped 1 previous similar message [ 1750.481792] Lustre: server umount lustre-MDT0000 complete [ 1754.506295] LustreError: 58177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788685561 with bad export cookie 9380230947271850554 [ 1754.511644] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1754.527297] LustreError: 58177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1755.071445] Lustre: server umount lustre-MDT0001 complete [ 1767.082077] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1769.545482] Key type lgssc unregistered [ 1769.890327] LNet: 61763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.907298] LNetError: 61763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.917283] LNet: Removed LNI 192.168.203.160@tcp [ 1770.774242] Key type .llcrypt unregistered [ 1770.776652] Key type ._llcrypt unregistered [ 1822.593059] Key type ._llcrypt registered [ 1822.599865] Key type .llcrypt registered [ 1822.728958] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1824.091350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.127392] alg: No test for adler32 (adler32-zlib) [ 1825.249168] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 1825.569979] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1827.344160] Key type lgssc registered [ 1828.560855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1832.776747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1832.801632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1834.114567] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1834.144366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1834.311024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1835.181281] Lustre: Failing over lustre-MDT0000 [ 1835.357187] Lustre: server umount lustre-MDT0000 complete [ 1842.587365] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 05:07:26 (1788685646) [ 1850.069885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1850.442830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.602622] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1862.005327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.147965] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1862.202995] Lustre: 64557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1862.506437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1866.244834] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1874.587584] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1879.765800] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1886.149441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1886.392729] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1886.624257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.167217] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1895.001304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1901.900558] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1908.300937] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1915.363214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1915.370459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1915.385155] Lustre: Skipped 2 previous similar messages [ 1915.394957] Lustre: Skipped 1 previous similar message [ 1920.485786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1920.488183] Lustre: Skipped 2 previous similar messages [ 1920.819832] Lustre: server umount lustre-MDT0000 complete [ 1923.824660] LustreError: 64550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788685730 with bad export cookie 12041113953458559369 [ 1923.834747] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1923.844688] LustreError: 64550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1924.110805] Lustre: server umount lustre-MDT0001 complete [ 1927.454208] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1931.280479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1939.756932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1941.984260] Lustre: 63482:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788685732/real 1788685732] req@0000000085494a63 x1875572825301248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788685748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1942.027840] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.058245] Lustre: 63479:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788685735/real 1788685735] req@000000002ca70ed5 x1875572825301376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788685751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1947.104148] Lustre: 63480:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788685737/real 1788685737] req@0000000063318f13 x1875572825301504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788685753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1949.216982] LustreError: 63478:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b8d7699b x1875572825302272/t0(0) o250->MGC192.168.203.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1949.714142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1949.811226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1950.754433] Lustre: 63482:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788685740/real 1788685740] req@000000002c68dc01 x1875572825301760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788685756 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1954.293748] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1954.784548] Lustre: 63482:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788685745/real 1788685745] req@000000005b63de2e x1875572825301888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788685761 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1956.837576] LustreError: 68330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1956.871489] LustreError: 68330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 1961.955548] LustreError: 68332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1961.968559] LustreError: 68332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1962.336501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.602540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1967.377829] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1968.110484] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1971.178771] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1974.607389] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1980.413144] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1988.070755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1988.084448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.098321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1988.107169] Lustre: Skipped 1 previous similar message [ 1991.649163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1991.676583] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.765902] Lustre: server umount lustre-MDT0000 complete [ 1996.782810] LustreError: 66406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.802577] LustreError: 66406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1997.333746] LustreError: 64550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788685803 with bad export cookie 12041113953458561644 [ 1997.336125] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1997.363107] LustreError: 64550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1997.683821] Lustre: server umount lustre-MDT0001 complete [ 2003.401455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2013.644275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.272185] Lustre: 63480:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788685808/real 1788685808] req@00000000d1d465ff x1875572825329664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788685824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2018.296156] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.304265] Lustre: Skipped 1 previous similar message [ 2022.988884] LustreError: 66407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2023.093782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2023.189482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2027.105031] LustreError: 66407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.120645] LustreError: 66407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2027.517282] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2035.551206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.248914] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2041.322402] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2047.464842] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2050.524863] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2057.937857] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2066.924707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2066.933053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2066.946172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2066.951927] Lustre: Skipped 2 previous similar messages [ 2070.892365] Lustre: server umount lustre-MDT0000 complete [ 2073.059223] LustreError: 71014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.073075] LustreError: 71014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2074.268745] LustreError: 64550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788685880 with bad export cookie 12041113953458563023 [ 2074.274318] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2074.282491] LustreError: 64550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2074.582930] Lustre: server umount lustre-MDT0001 complete [ 2080.787848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2084.099695] Lustre: DEBUG MARKER: files compared the same [ 2096.316134] Lustre: server umount lustre-OST0000 complete [ 2112.592198] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2115.790086] Key type lgssc unregistered [ 2116.137965] LNet: 74288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2116.149824] LNetError: 74288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.164871] LNet: Removed LNI 192.168.203.160@tcp [ 2117.000633] Key type .llcrypt unregistered [ 2117.004587] Key type ._llcrypt unregistered [ 2141.753083] Key type ._llcrypt registered [ 2141.758154] Key type .llcrypt registered [ 2141.877845] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2143.314273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2143.346583] alg: No test for adler32 (adler32-zlib) [ 2144.442599] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 2144.684625] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2146.353546] Key type lgssc registered [ 2147.406714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2154.233666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2154.254696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2155.683281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2159.341799] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug +malloc all [ 2166.244480] LustreError: 75701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.274243] LustreError: 75701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2166.674376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2166.987436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2171.287630] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug +malloc all [ 2178.930627] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2185.547907] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2192.766441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2193.114213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.284600] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug +malloc all [ 2201.328899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2206.775715] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2222.057574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.071841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2222.088936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2227.169390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2227.173848] Lustre: Skipped 1 previous similar message [ 2232.288813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2232.300540] Lustre: Skipped 1 previous similar message [ 2233.312286] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2233.536746] Lustre: server umount lustre-OST0000 complete [ 2238.945102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2238.958484] LustreError: Skipped 1 previous similar message [ 2238.962797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.983275] Lustre: Skipped 1 previous similar message [ 2238.987452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2238.990201] Lustre: Skipped 1 previous similar message [ 2242.551043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.705525] Lustre: server umount lustre-MDT0000 complete [ 2247.541215] LustreError: 77545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686054 with bad export cookie 16965110498480155471 [ 2247.548642] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2247.551847] LustreError: 77545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2247.892565] Lustre: server umount lustre-MDT0001 complete [ 2257.703653] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2260.099442] Key type lgssc unregistered [ 2260.362434] LNet: 79248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2260.369525] LNetError: 79248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2260.385329] LNet: Removed LNI 192.168.203.160@tcp [ 2261.098808] Key type .llcrypt unregistered [ 2261.104745] Key type ._llcrypt unregistered [ 2281.041746] Key type ._llcrypt registered [ 2281.043637] Key type .llcrypt registered [ 2281.177949] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2281.941714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2281.960608] alg: No test for adler32 (adler32-zlib) [ 2282.978390] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 2283.228122] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2284.992258] Key type lgssc registered [ 2285.991493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2293.051260] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2293.090424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2308.640485] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2313.760789] LustreError: 80615:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 2340.385088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2345.020403] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2351.731862] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2358.338831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.461914] Lustre: *** cfs_fail_loc=706, val=0*** [ 2358.872644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.957111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2362.251944] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2363.889341] LustreError: 80641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.909973] LustreError: 80641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2368.998274] LustreError: 80640:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.836666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2370.093057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2373.167117] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2379.604392] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2385.482370] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2389.480153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2389.487738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2389.499756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2390.504621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2390.512573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2390.521819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2391.527215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2393.878188] Lustre: server umount lustre-OST0000 complete [ 2400.749993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2400.760131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2400.781638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2400.797789] Lustre: Skipped 1 previous similar message [ 2404.256872] Lustre: server umount lustre-MDT0000 complete [ 2406.885071] LustreError: 81706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.921634] LustreError: 81706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2407.182973] LustreError: 80623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686213 with bad export cookie 660664850743260435 [ 2407.191087] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2407.204704] LustreError: 80623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2407.673868] Lustre: server umount lustre-MDT0001 complete [ 2417.548263] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2420.157249] Key type lgssc unregistered [ 2420.427930] LNet: 84230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2420.434606] LNetError: 84230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.446801] LNet: Removed LNI 192.168.203.160@tcp [ 2421.240720] Key type .llcrypt unregistered [ 2421.243234] Key type ._llcrypt unregistered [ 2443.880449] Key type ._llcrypt registered [ 2443.888843] Key type .llcrypt registered [ 2443.999905] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2444.832320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2444.854077] alg: No test for adler32 (adler32-zlib) [ 2445.942047] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 2446.179263] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2447.840165] Key type lgssc registered [ 2448.836769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2455.597538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2455.616138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.335475] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2465.996263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.415277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2469.709063] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2471.911114] LustreError: 86327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2471.930353] LustreError: 86327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2476.024352] LustreError: 86326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.039369] LustreError: 86326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2476.748485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2477.016393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.150615] LustreError: 86331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.169896] LustreError: 86331:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2482.358092] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2487.269847] LustreError: 86325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.284629] LustreError: 86325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2489.906356] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2489.918461] Lustre: MGS: Client 989bfa14-a8df-4f0f-a3aa-165f366e1819 (at 0@lo) reconnecting [ 2489.930139] LustreError: 86341:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2490.114849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2490.187078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2494.058837] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2506.729281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2506.739162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2506.752526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2506.752754] Lustre: Skipped 1 previous similar message [ 2506.782920] Lustre: Skipped 1 previous similar message [ 2511.843704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2511.849117] Lustre: Skipped 1 previous similar message [ 2512.642083] Lustre: server umount lustre-OST0000 complete [ 2516.961936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.971250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2516.973792] Lustre: Skipped 1 previous similar message [ 2516.986034] Lustre: Skipped 1 previous similar message [ 2521.836915] Lustre: server umount lustre-MDT0000 complete [ 2522.082104] LustreError: 86330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.095522] LustreError: 86330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2525.890937] Lustre: server umount lustre-MDT0001 complete [ 2530.554972] Lustre: server umount MGS complete [ 2541.654902] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 05:19:06 (1788686346) [ 2567.059995] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 2574.781234] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2603.601859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2607.438897] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2616.307686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2616.490452] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2616.524566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2616.625569] Lustre: lustre-MDT0001: new disk, initializing [ 2616.699187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2616.725484] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2616.730138] Lustre: 93494:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2616.735140] LustreError: 93494:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2620.056542] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2629.491194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2629.730852] Lustre: lustre-OST0000: new disk, initializing [ 2629.734895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2629.744552] Lustre: 94261:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2629.804036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.291195] Lustre: 94353:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005310ef4a x1875573475416064/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788686454 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2635.392785] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2644.559301] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2650.841655] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2650.854690] Lustre: MGS: Client 683b3381-c163-4c40-8ed2-75fde660dfbb (at 0@lo) reconnecting [ 2650.912186] Lustre: 95307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2650.927451] LustreError: 95307:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2650.936955] LustreError: 95307:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2650.976416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2651.085118] Lustre: lustre-MDT0000: new disk, initializing [ 2651.241585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.262073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2651.284463] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2652.324296] Lustre: 95331:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ac0b9ccf x1875573475423104/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788686474 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2655.201982] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2657.442729] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2661.422032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2661.432264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2661.511185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2685.414956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2685.419102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2685.425481] Lustre: Skipped 1 previous similar message [ 2690.545655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2690.556480] Lustre: Skipped 2 previous similar messages [ 2698.208122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2698.372589] Lustre: server umount lustre-OST0000 complete [ 2705.893954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2705.912938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2705.916747] Lustre: Skipped 3 previous similar messages [ 2707.849967] Lustre: server umount lustre-MDT0000 complete [ 2711.012821] LustreError: 93469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.045058] LustreError: 93469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2726.368117] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2726.555479] Lustre: server umount lustre-MDT0001 complete [ 2729.670612] Lustre: server umount MGS complete [ 2740.061794] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 05:22:24 (1788686544) [ 2759.328996] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2762.801966] Key type lgssc unregistered [ 2763.266123] LNet: 97833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2763.286304] LNetError: 97833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2763.346329] LNet: Removed LNI 192.168.203.160@tcp [ 2764.347156] Key type .llcrypt unregistered [ 2764.349496] Key type ._llcrypt unregistered [ 2781.771739] Key type ._llcrypt registered [ 2781.773667] Key type .llcrypt registered [ 2781.854678] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2782.706677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2782.760152] alg: No test for adler32 (adler32-zlib) [ 2783.820533] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 2783.994400] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2785.664139] Key type lgssc registered [ 2786.569836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2793.333922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2793.348259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.726360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2798.562598] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2805.219019] LustreError: 99053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.251732] LustreError: 99053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2806.463860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2806.801841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2811.377313] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2817.414601] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2822.681824] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2829.518188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2829.903143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2835.436047] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2840.196747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2844.170568] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2855.907440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2855.923099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2855.932447] Lustre: Skipped 1 previous similar message [ 2859.706324] Lustre: server umount lustre-OST0000 complete [ 2866.145761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.148817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2866.177795] Lustre: Skipped 1 previous similar message [ 2866.197401] Lustre: Skipped 2 previous similar messages [ 2870.493222] Lustre: server umount lustre-MDT0000 complete [ 2871.265558] LustreError: 99047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2871.283442] LustreError: 99047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2873.953562] LustreError: 99033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686680 with bad export cookie 9318458149542049624 [ 2873.956692] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2873.961644] LustreError: 99033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2874.190717] Lustre: server umount lustre-MDT0001 complete [ 2884.193939] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2887.037962] Key type lgssc unregistered [ 2887.290578] LNet: 102583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2887.297770] LNetError: 102583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2887.306665] LNet: Removed LNI 192.168.203.160@tcp [ 2887.984826] Key type .llcrypt unregistered [ 2887.990325] Key type ._llcrypt unregistered [ 2903.692555] Key type ._llcrypt registered [ 2903.695279] Key type .llcrypt registered [ 2903.871514] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2904.763046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2904.800302] alg: No test for adler32 (adler32-zlib) [ 2905.930583] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 2906.117947] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2907.784316] Key type lgssc registered [ 2909.072379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2915.351310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2915.357337] LustreError: 103663:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 2915.359258] Lustre: Skipped 1 previous similar message [ 2915.359275] LustreError: 103662:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 2915.381864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2915.383645] LustreError: 103663:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 2915.383672] LustreError: 103663:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2915.383692] LustreError: 103663:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2916.952755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.060224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2924.283205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2928.413734] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2933.811095] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing lsmod [ 2936.851460] LustreError: 104864:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 2936.923838] LustreError: 104871:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 2936.931789] LustreError: 104864:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4919 [ 2936.938025] LustreError: 104864:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2936.950596] LustreError: 104864:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2936.955242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2937.350258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2942.438141] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.444866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.457168] Lustre: Skipped 2 previous similar messages [ 2942.470631] Lustre: Skipped 1 previous similar message [ 2946.434424] Lustre: server umount lustre-MDT0000 complete [ 2947.554384] LustreError: 103701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.582353] LustreError: 103701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2948.582211] LustreError: 103695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2948.608565] LustreError: 103695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2949.643613] LustreError: 103677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686756 with bad export cookie 11155830009557135444 [ 2949.648705] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2949.653548] LustreError: 103677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2949.677750] Lustre: 103692:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2949.985380] Lustre: server umount lustre-MDT0001 complete [ 2964.730543] Lustre: server umount lustre-OST0000 complete [ 2973.751675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.209095] LustreError: 105935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2974.259873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2978.772956] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2979.297842] LustreError: 105936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2984.424530] LustreError: 105935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2988.101917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2988.510486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2993.504973] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3000.414637] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3006.517363] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3013.021740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3013.374611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3018.501403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3018.942638] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3026.419850] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3039.202906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3039.218594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3039.255240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3039.268786] Lustre: Skipped 1 previous similar message [ 3043.299132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3043.308654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3043.326922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3044.873454] Lustre: server umount lustre-OST0000 complete [ 3049.955614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3049.963184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3049.978553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3049.983861] Lustre: Skipped 1 previous similar message [ 3058.658866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3058.667265] Lustre: Skipped 3 previous similar messages [ 3063.265829] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3063.394339] Lustre: server umount lustre-MDT0000 complete [ 3063.778090] LustreError: 105936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3063.788547] LustreError: 105936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3067.201653] LustreError: 105916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686873 with bad export cookie 11155830009557136123 [ 3067.214502] LustreError: 105916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3067.217943] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3067.569652] Lustre: server umount lustre-MDT0001 complete [ 3079.947237] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 3082.848967] Key type lgssc unregistered [ 3083.141170] LNet: 109482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3083.152809] LNetError: 109482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3084.198042] LNet: Removed LNI 192.168.203.160@tcp [ 3085.175339] Key type .llcrypt unregistered [ 3085.177270] Key type ._llcrypt unregistered [ 3111.020580] Key type ._llcrypt registered [ 3111.023801] Key type .llcrypt registered [ 3111.144401] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3112.385385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3112.449557] alg: No test for adler32 (adler32-zlib) [ 3113.532688] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 3113.815579] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 3115.544180] Key type lgssc registered [ 3116.651740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3124.492941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3124.520720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.195682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3131.455934] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3135.466377] LustreError: 110902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3135.500692] LustreError: 110902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3140.589527] LustreError: 110903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3141.041911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3141.400497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3146.053294] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3154.422492] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3159.600771] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3166.373319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3166.639329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3170.762099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3173.390975] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3181.257221] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3190.822617] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3194.753444] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3201.506571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3201.515577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3201.516405] LustreError: Skipped 1 previous similar message [ 3201.517167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3201.548830] Lustre: Skipped 1 previous similar message [ 3205.495415] Lustre: server umount lustre-OST0000 complete [ 3211.749192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3211.750476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3211.766357] Lustre: Skipped 1 previous similar message [ 3211.796986] Lustre: Skipped 2 previous similar messages [ 3215.699081] Lustre: server umount lustre-MDT0000 complete [ 3216.874535] LustreError: 110902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3216.909351] LustreError: 110902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3220.019558] LustreError: 110882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788687026 with bad export cookie 3103149517194414820 [ 3220.030616] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3220.039776] LustreError: 110882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3220.404796] Lustre: server umount lustre-MDT0001 complete [ 3231.778738] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 3234.528416] Key type lgssc unregistered [ 3234.820887] LNet: 114550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3234.826967] LNetError: 114550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3234.842236] LNet: Removed LNI 192.168.203.160@tcp [ 3235.605420] Key type .llcrypt unregistered [ 3235.608156] Key type ._llcrypt unregistered [ 3253.601201] Key type ._llcrypt registered [ 3253.603080] Key type .llcrypt registered [ 3253.650302] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3254.391142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3254.404195] alg: No test for adler32 (adler32-zlib) [ 3255.327416] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 3255.502549] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 3257.160293] Key type lgssc registered [ 3258.163978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3265.814879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3265.827394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3267.432124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3272.807076] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3276.772993] LustreError: 115772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.794825] LustreError: 115772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3281.889970] LustreError: 115773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3282.426327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3283.058538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3288.346918] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3297.107149] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3303.668900] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3311.920555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3312.048049] LustreError: 117594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.339782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3317.752132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3319.511271] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3327.478830] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3338.213677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3338.226077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3338.244307] Lustre: Skipped 1 previous similar message [ 3338.275703] Lustre: Skipped 1 previous similar message [ 3343.358793] Lustre: server umount lustre-OST0000 complete [ 3348.449461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3348.456696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3348.463830] Lustre: Skipped 1 previous similar message [ 3353.570819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3353.586423] Lustre: Skipped 2 previous similar messages [ 3354.507253] Lustre: server umount lustre-MDT0000 complete [ 3357.666870] LustreError: 115772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.696164] LustreError: 115772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3358.165609] LustreError: 115751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788687164 with bad export cookie 4100491468100893431 [ 3358.180159] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3358.193373] LustreError: 115751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3358.577888] Lustre: server umount lustre-MDT0001 complete [ 3368.458818] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 3371.312971] Key type lgssc unregistered [ 3371.575498] LNet: 119305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3371.587701] LNetError: 119305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3371.604685] LNet: Removed LNI 192.168.203.160@tcp [ 3372.342880] Key type .llcrypt unregistered [ 3372.345034] Key type ._llcrypt unregistered [ 3399.048850] Key type ._llcrypt registered [ 3399.050899] Key type .llcrypt registered [ 3399.207620] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3400.392334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3400.421318] alg: No test for adler32 (adler32-zlib) [ 3401.730537] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 3402.182922] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 3403.912526] Key type lgssc registered [ 3405.294841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3414.640649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3414.671758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3416.295745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.033317] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3425.769559] LustreError: 120719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3425.784923] LustreError: 120719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3430.034410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3430.411977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3435.419800] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3442.966270] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3449.088150] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3456.109268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3456.508583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3460.609508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3462.843703] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3471.318323] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3482.265676] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3486.185522] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3486.195099] Lustre: Skipped 2 previous similar messages [ 3490.726268] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3490.743556] Lustre: Skipped 2 previous similar messages [ 3490.909517] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3490.977788] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3497.975497] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.60@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3497.982647] Lustre: Skipped 2 previous similar messages [ 3499.554467] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.60@tcp 192.168.0.[2,10]@tcp [ 3505.442692] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.60/24@tcp in log lustre-MDT0000 [ 3505.451140] Lustre: Skipped 2 previous similar messages [ 3507.994757] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.60/24@tcp [ 3508.002709] Lustre: Skipped 1 previous similar message [ 3513.834381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3513.838485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.850123] LustreError: Skipped 1 previous similar message [ 3513.882763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3515.876804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3515.881210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3515.893759] LustreError: Skipped 4 previous similar messages [ 3515.896829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3520.995922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3521.004185] Lustre: Skipped 2 previous similar messages [ 3526.128718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3526.132305] Lustre: Skipped 1 previous similar message [ 3528.160687] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3528.396714] Lustre: server umount lustre-OST0000 complete [ 3532.773176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3532.785690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.809680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3536.356165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3538.409781] Lustre: server umount lustre-MDT0000 complete [ 3541.478047] LustreError: 122763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3541.491532] LustreError: 122763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3541.851896] LustreError: 120699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788687348 with bad export cookie 11273604210864290115 [ 3541.863884] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3541.867815] LustreError: 120699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3542.195966] Lustre: server umount lustre-MDT0001 complete [ 3553.697978] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 3556.767369] Key type lgssc unregistered [ 3557.066057] LNet: 125366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3557.084444] LNetError: 125366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3558.123314] LNet: Removed LNI 192.168.203.160@tcp [ 3559.042437] Key type .llcrypt unregistered [ 3559.048302] Key type ._llcrypt unregistered [ 3585.988733] Key type ._llcrypt registered [ 3585.991250] Key type .llcrypt registered [ 3586.088533] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3586.999653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3587.134353] alg: No test for adler32 (adler32-zlib) [ 3588.183069] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 3588.467681] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 3590.152447] Key type lgssc registered [ 3591.344498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3599.184462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3599.203720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3600.726506] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3600.750759] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.60/24@tcp [ 3600.773560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3605.579633] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3611.115285] LustreError: 126925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3611.140307] LustreError: 126925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3614.330972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.724344] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3614.757215] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.60/24@tcp [ 3614.845513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3620.347192] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3628.724048] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3635.336546] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3643.940326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3644.393602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3646.484780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3651.169436] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3658.536540] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3672.038712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3672.048336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3672.061727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3676.132798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3676.135144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3676.144242] LustreError: Skipped 1 previous similar message [ 3676.158566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.410306] Lustre: server umount lustre-OST0000 complete [ 3681.251120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3681.251395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3681.266904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3681.282639] Lustre: Skipped 2 previous similar messages [ 3686.371151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3686.374162] Lustre: Skipped 1 previous similar message [ 3687.508697] Lustre: server umount lustre-MDT0000 complete [ 3691.302273] LustreError: 126906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788687497 with bad export cookie 534754011811245426 [ 3691.313968] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3691.321669] LustreError: 126906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3691.734032] Lustre: server umount lustre-MDT0001 complete [ 3703.591327] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 3706.719970] Key type lgssc unregistered [ 3707.054514] LNet: 130511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3707.062719] LNetError: 130511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3707.085423] LNet: Removed LNI 192.168.203.160@tcp [ 3708.024910] Key type .llcrypt unregistered [ 3708.034140] Key type ._llcrypt unregistered [ 3736.447666] Key type ._llcrypt registered [ 3736.448802] Key type .llcrypt registered [ 3736.557863] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 3754.348299] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3755.778464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3755.939449] alg: No test for adler32 (adler32-zlib) [ 3757.256308] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 3757.499369] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 3759.176229] Key type lgssc registered [ 3760.398868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3799.985832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3799.999552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.394495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3801.496743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3801.642264] Lustre: lustre-MDT0000: new disk, initializing [ 3801.826556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.852562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3806.440699] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3817.641072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.834886] Lustre: 133934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3817.920956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3817.926247] Lustre: Skipped 1 previous similar message [ 3818.041943] Lustre: lustre-MDT0001: new disk, initializing [ 3818.159237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3818.225645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3818.247815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3823.223400] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3833.704332] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3840.274387] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3847.016754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3847.235534] Lustre: lustre-OST0000: new disk, initializing [ 3847.240909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3847.247483] Lustre: 135901:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3847.322234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3853.952947] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3856.433349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3856.445417] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3856.526241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3863.169923] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3867.948478] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3868.158306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3872.685711] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3872.887439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3877.346123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3877.364201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3877.380709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3882.480545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3882.486220] Lustre: Skipped 2 previous similar messages [ 3887.586816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3887.596470] Lustre: Skipped 1 previous similar message [ 3889.120120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3889.299828] Lustre: server umount lustre-OST0000 complete [ 3895.267975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3895.275279] LustreError: Skipped 1 previous similar message [ 3895.277980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3895.285801] Lustre: Skipped 1 previous similar message [ 3895.289057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3897.826572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3899.288321] Lustre: server umount lustre-MDT0000 complete [ 3903.640688] LustreError: 133923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788687710 with bad export cookie 4692974548268620821 [ 3903.670580] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3903.677065] LustreError: 133923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3904.265569] Lustre: server umount lustre-MDT0001 complete [ 3914.532168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.935488] LustreError: 138121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3914.954987] LustreError: 138121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3915.015908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3920.297408] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3920.360510] LustreError: 138122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3924.451879] LustreError: 138121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3929.578053] LustreError: 138122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3930.494974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.940800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3935.809309] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3943.326932] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3949.616326] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3956.096791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3956.417405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3963.194856] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3972.205969] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3977.368920] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3977.643104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3982.437170] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3982.713782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3991.562166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3991.805963] Lustre: lustre-OST0001: new disk, initializing [ 3991.810231] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3991.822350] Lustre: 141491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3991.929771] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3999.355769] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4001.895839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4001.905923] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4001.984295] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4008.572368] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4008.756692] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4012.921551] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4013.113660] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4019.405895] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c9f837da800.ost_server_uuid 50 [ 4021.581577] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c9f837da800.ost_server_uuid in IDLE state after 0 sec [ 4038.113713] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4038.118877] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4038.127455] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4038.134868] Lustre: Skipped 2 previous similar messages [ 4040.850526] Lustre: server umount lustre-OST0001 complete [ 4043.239317] LustreError: 139943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.292799] LustreError: 139943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4046.830131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4046.840401] LustreError: Skipped 1 previous similar message [ 4046.847310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.863305] Lustre: Skipped 1 previous similar message [ 4046.871531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4051.256937] Lustre: server umount lustre-OST0000 complete [ 4058.603207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4058.618686] Lustre: Skipped 1 previous similar message [ 4061.769951] Lustre: server umount lustre-MDT0000 complete [ 4063.713784] LustreError: 138121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4063.731541] LustreError: 138121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4065.235231] LustreError: 138103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788687871 with bad export cookie 4692974548268621703 [ 4065.240898] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4065.249066] LustreError: 138103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4065.658908] Lustre: server umount lustre-MDT0001 complete [ 4085.533652] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 4088.412350] Key type lgssc unregistered [ 4088.734758] LNet: 144521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4088.743510] LNetError: 144521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4088.765076] LNet: Removed LNI 192.168.203.160@tcp [ 4089.808785] Key type .llcrypt unregistered [ 4089.811119] Key type ._llcrypt unregistered [ 4128.398420] Key type ._llcrypt registered [ 4128.401026] Key type .llcrypt registered [ 4128.500529] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 4141.663618] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 4142.946454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4142.997866] alg: No test for adler32 (adler32-zlib) [ 4144.092188] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 4144.290633] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 4145.968176] Key type lgssc registered [ 4147.087258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4180.983097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4181.014122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4182.249230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4182.272190] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4182.350335] Lustre: lustre-MDT0000: new disk, initializing [ 4182.458891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4182.484457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4186.963328] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4198.975727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4199.053517] Lustre: 148652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4199.097531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4199.100931] Lustre: Skipped 1 previous similar message [ 4199.145836] Lustre: lustre-MDT0001: new disk, initializing [ 4199.219365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4199.261122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4199.279506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4204.014507] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4213.317812] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4219.347277] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4226.198047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4226.572284] Lustre: lustre-OST0000: new disk, initializing [ 4226.575660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4226.584187] Lustre: 150622:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4226.672612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4232.868726] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4235.324990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4235.335140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4235.439114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4241.890889] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4247.159699] Lustre: Failing over lustre-OST0000 [ 4247.337762] Lustre: server umount lustre-OST0000 complete [ 4265.441464] Lustre: 146874:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788688055/real 1788688055] req@0000000000025c1d x1875575255892736/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788688071 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4265.467217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4265.577970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4265.946950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.967920] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4266.531046] Lustre: 146876:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788688057/real 1788688057] req@00000000e16a8ce3 x1875575255892992/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788688073 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4266.554780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.571322] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4267.616235] Lustre: 146877:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788688057/real 1788688057] req@000000006108067f x1875575255893120/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788688073 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4271.648928] Lustre: 146877:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788688062/real 1788688062] req@000000006b7bb533 x1875575255895040/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788688078 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4271.698040] Lustre: 146877:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4271.781597] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4275.781236] Lustre: lustre-OST0000: Recovery over after 0:09, of 3 clients 3 recovered and 0 were evicted. [ 4275.783917] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4276.027956] Lustre: Failing over lustre-MDT0000 [ 4276.194558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4276.201046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4276.209746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4276.423435] Lustre: server umount lustre-MDT0000 complete [ 4277.217525] Lustre: 146876:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788688067/real 1788688067] req@000000003ef66d10 x1875575255896064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788688083 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4277.254207] Lustre: 146876:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4280.804362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4280.813976] Lustre: Skipped 1 previous similar message [ 4285.601663] LustreError: 148657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4285.632784] LustreError: 148657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4290.725939] LustreError: 148657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4290.758976] LustreError: 148657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4295.849549] LustreError: 148659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4295.869553] LustreError: 148659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4295.947955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4296.117318] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4296.382815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4296.430866] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4296.736845] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4301.174229] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4301.836483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4301.862415] Lustre: Skipped 1 previous similar message [ 4301.901600] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4301.974259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4312.038356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4312.046210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4312.054638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4312.077285] Lustre: Skipped 1 previous similar message [ 4312.090715] Lustre: Skipped 1 previous similar message [ 4315.612711] Lustre: server umount lustre-OST0000 complete [ 4322.273472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4322.275469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4322.287774] Lustre: Skipped 1 previous similar message [ 4322.303171] Lustre: Skipped 1 previous similar message [ 4325.292457] Lustre: server umount lustre-MDT0000 complete [ 4327.393125] LustreError: 151543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4327.434571] LustreError: 151543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4329.021675] LustreError: 148643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788688135 with bad export cookie 12666949189667084373 [ 4329.025238] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4329.034182] LustreError: 148643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4329.456196] Lustre: server umount lustre-MDT0001 complete [ 4340.837797] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 4343.576271] Key type lgssc unregistered [ 4343.960749] LNet: 154275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4343.966937] LNetError: 154275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4343.981330] LNet: Removed LNI 192.168.203.160@tcp [ 4344.905872] Key type .llcrypt unregistered [ 4344.909347] Key type ._llcrypt unregistered [ 4381.088494] Key type ._llcrypt registered [ 4381.092876] Key type .llcrypt registered [ 4381.204119] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 4382.387877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4382.547109] alg: No test for adler32 (adler32-zlib) [ 4383.561613] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 4383.764147] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 4385.424631] Key type lgssc registered [ 4386.334538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4392.463958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4392.483340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4393.959727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4397.605192] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4404.199503] LustreError: 155793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.223730] LustreError: 155793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4407.749782] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4409.317468] LustreError: 155794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4411.820866] LustreError: 155822:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4411.831484] LustreError: 155822:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4411.840840] LustreError: 155822:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4412.229146] Lustre: server umount lustre-MDT0000 complete [ 4428.084905] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 4438.680766] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 4475.573959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4475.801635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4475.845547] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4475.938439] Lustre: lustre-MDT0000: new disk, initializing [ 4476.006819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4476.024121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4480.825144] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4491.050652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4491.197990] Lustre: 159126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4491.249289] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4491.257976] Lustre: Skipped 1 previous similar message [ 4491.356173] Lustre: lustre-MDT0001: new disk, initializing [ 4491.427611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4491.451974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4491.473729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4496.709613] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4507.802554] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4515.102150] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4521.711332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4521.921878] Lustre: lustre-OST0000: new disk, initializing [ 4521.925321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4521.930639] Lustre: 161093:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4522.002237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4527.557947] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4531.279979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4531.304322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4531.433415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4538.068746] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4543.121978] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4545.354831] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 05:52:29 (1788688349) [ 4795.873247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4795.876323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4795.894637] Lustre: Skipped 2 previous similar messages [ 4795.918404] Lustre: Skipped 2 previous similar messages [ 4801.003793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4801.013841] Lustre: Skipped 2 previous similar messages [ 4801.539788] Lustre: server umount lustre-MDT0000 complete [ 4805.003453] LustreError: 159117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788688611 with bad export cookie 1465342430778887064 [ 4805.014219] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4806.114259] LustreError: 162858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4806.114321] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4806.159992] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4810.150263] LustreError: 159131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4810.150354] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4811.358331] Lustre: server umount lustre-MDT0001 complete [ 4821.299846] Lustre: server umount lustre-OST0000 complete [ 4831.428446] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 4842.547164] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 4880.710917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4881.010719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4881.097551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4881.226973] Lustre: lustre-MDT0000: new disk, initializing [ 4881.370545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4881.407342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4887.238750] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4898.080837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4898.185542] Lustre: 165762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4898.237909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4898.242196] Lustre: Skipped 1 previous similar message [ 4898.288899] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4898.347533] Lustre: lustre-MDT0001: new disk, initializing [ 4898.422552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4898.452555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4898.467286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4902.796747] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4912.501238] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4918.367786] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4925.057746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4925.329687] Lustre: lustre-OST0000: new disk, initializing [ 4925.332805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4925.350217] Lustre: 167730:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4925.453886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4931.430500] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4932.711360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4932.726695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4932.829936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4940.611870] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4945.630511] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4945.827561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4950.697853] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4950.970629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4953.581095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4953.577424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4953.585781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4953.603071] Lustre: Skipped 1 previous similar message [ 4953.635232] LustreError: Skipped 1 previous similar message [ 4967.392127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4967.588057] Lustre: server umount lustre-OST0000 complete [ 4974.050023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4974.060769] Lustre: Skipped 1 previous similar message [ 4974.062538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4974.071637] Lustre: Skipped 6 previous similar messages [ 4977.825086] Lustre: server umount lustre-MDT0000 complete [ 4979.185904] LustreError: 166882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4981.819746] LustreError: 165753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788688788 with bad export cookie 1465342430779045894 [ 4981.830197] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4981.846320] LustreError: 165753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4982.077065] Lustre: server umount lustre-MDT0001 complete [ 4992.161050] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:59:56 (1788688796) [ 5009.666447] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 5019.462475] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5058.035978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5058.246834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5058.295531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5058.403876] Lustre: lustre-MDT0000: new disk, initializing [ 5058.538672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5058.574359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5063.384678] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5073.988791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.139844] Lustre: 172498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5074.489962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5078.911639] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5088.725808] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5094.820887] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5101.371187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5101.750420] Lustre: lustre-OST0000: new disk, initializing [ 5101.753857] Lustre: Skipped 1 previous similar message [ 5101.756814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5101.760568] Lustre: Skipped 2 previous similar messages [ 5101.765229] Lustre: 174477:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5108.252689] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5108.810953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5108.820102] Lustre: Skipped 1 previous similar message [ 5108.826147] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5108.903599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5118.384832] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5129.707968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5129.716571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5129.718912] LustreError: Skipped 1 previous similar message [ 5129.740232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5129.766013] Lustre: Skipped 1 previous similar message [ 5132.628369] Lustre: server umount lustre-OST0000 complete [ 5139.937201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5139.951793] Lustre: Skipped 1 previous similar message [ 5142.922526] Lustre: server umount lustre-MDT0000 complete [ 5145.063410] LustreError: 172510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5145.095078] LustreError: 172510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5147.719246] LustreError: 174476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788688954 with bad export cookie 1465342430779046776 [ 5147.724651] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5147.731465] LustreError: 174476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5148.172176] Lustre: server umount lustre-MDT0001 complete [ 5161.795659] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:02:44 (1788688964) [ 5183.035429] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 5195.005958] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5232.615526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5232.865473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5232.899485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5233.018918] Lustre: lustre-MDT0000: new disk, initializing [ 5233.131586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5233.137143] Lustre: Skipped 2 previous similar messages [ 5233.160832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5237.613646] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5250.517039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5250.635500] Lustre: 178802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5250.931530] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5256.082738] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5266.286457] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5272.567432] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5279.880633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5280.303763] Lustre: 180779:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5286.683220] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5289.023210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5289.140374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5296.318415] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5304.292509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5304.301978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5304.312845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5304.320946] Lustre: Skipped 3 previous similar messages [ 5309.577433] Lustre: server umount lustre-OST0000 complete [ 5327.328140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5327.465677] Lustre: server umount lustre-MDT0000 complete [ 5330.411888] LustreError: 178811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5330.433540] LustreError: 178811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5330.541541] LustreError: 181883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689137 with bad export cookie 1465342430779047861 [ 5330.545804] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5330.551619] LustreError: 181883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5330.894711] Lustre: server umount lustre-MDT0001 complete [ 5341.621847] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5344.038040] Key type lgssc unregistered [ 5344.298092] LNet: 182635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5344.310651] LNetError: 182635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5344.327701] LNet: Removed LNI 192.168.203.160@tcp [ 5345.106771] Key type .llcrypt unregistered [ 5345.110835] Key type ._llcrypt unregistered [ 5372.959899] Key type ._llcrypt registered [ 5372.967734] Key type .llcrypt registered [ 5373.164626] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5374.737877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5374.768213] alg: No test for adler32 (adler32-zlib) [ 5375.824185] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 5376.127195] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 5377.984264] Key type lgssc registered [ 5379.325100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5388.294703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5388.334786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5389.853740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5394.226904] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5400.040145] LustreError: 184050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5400.054072] LustreError: 184050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5402.477946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5402.956375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5407.216657] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5414.525939] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5420.123822] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5426.651495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5426.981473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5432.835885] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5435.193133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5440.327478] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5460.965670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.967049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5460.970184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5460.981952] Lustre: Skipped 1 previous similar message [ 5465.612629] Lustre: server umount lustre-OST0000 complete [ 5469.666689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5469.671394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5469.687723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5469.697370] Lustre: Skipped 1 previous similar message [ 5471.205164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5471.205901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5475.275667] Lustre: server umount lustre-MDT0000 complete [ 5476.322191] LustreError: 186538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.351171] LustreError: 186538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5478.944669] LustreError: 184029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689285 with bad export cookie 2917084812539903817 [ 5478.954254] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5478.981574] LustreError: 184029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5479.392981] Lustre: server umount lustre-MDT0001 complete [ 5491.102020] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5493.482547] Key type lgssc unregistered [ 5493.787903] LNet: 187601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5493.797486] LNetError: 187601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5493.826124] LNet: Removed LNI 192.168.203.160@tcp [ 5494.582595] Key type .llcrypt unregistered [ 5494.584558] Key type ._llcrypt unregistered [ 5520.198721] Key type ._llcrypt registered [ 5520.200404] Key type .llcrypt registered [ 5520.365818] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5521.522216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5521.651873] alg: No test for adler32 (adler32-zlib) [ 5522.660795] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 5522.793700] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 5524.433544] Key type lgssc registered [ 5525.484288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5531.991806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5532.004292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5533.360721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5537.664237] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5542.890215] LustreError: 189016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5542.919656] LustreError: 189016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5546.322806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5546.686029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5551.245108] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5558.636472] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5563.536785] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5569.784875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5570.115224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5576.391921] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5578.293453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5583.744252] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5591.730498] Lustre: lustre-OST0000: Not available for connect from 192.168.203.60@tcp (stopping) [ 5592.546058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5592.559946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5592.571803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5593.572698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5593.581689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5593.606178] Lustre: Skipped 1 previous similar message [ 5596.769234] Lustre: server umount lustre-OST0000 complete [ 5604.968635] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9c9f890fe000.ost_server_uuid 50 [ 5606.631634] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5623.265495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5623.275612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5623.288633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5626.973197] Lustre: server umount lustre-MDT0000 complete [ 5629.410736] LustreError: 189015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.433510] LustreError: 189015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5630.229801] LustreError: 190861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689436 with bad export cookie 12131085259920116659 [ 5630.232651] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5630.241756] LustreError: 190861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5630.628631] Lustre: server umount lustre-MDT0001 complete [ 5639.469551] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 06:10:43 (1788689443) [ 5646.451175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5646.972845] LustreError: 192933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5647.087304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5651.516492] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5652.450480] LustreError: 192934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5652.466647] LustreError: 192934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5656.550283] LustreError: 192933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5659.532949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5659.895612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5665.034461] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5672.650382] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5677.232828] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5682.543377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5682.850979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5687.659349] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5692.087196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5694.022930] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5699.663441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5699.819430] Lustre: lustre-OST0001: new disk, initializing [ 5699.822887] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5699.827896] Lustre: 195773:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5699.920852] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5704.099995] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5706.292055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5706.304900] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5706.394414] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5712.245619] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5716.460170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5716.463314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5716.474868] Lustre: Skipped 2 previous similar messages [ 5716.489217] Lustre: Skipped 3 previous similar messages [ 5721.206142] Lustre: server umount lustre-OST0000 complete [ 5721.573801] LustreError: 194754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5721.610871] LustreError: 194754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5727.623312] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5727.838564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5731.591367] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5731.734759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5738.652923] LustreError: 196581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5738.680755] LustreError: 196581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5747.172447] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5747.204454] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5747.210843] Lustre: Skipped 2 previous similar messages [ 5751.632688] Lustre: server umount lustre-OST0001 complete [ 5757.408989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5757.426947] Lustre: Skipped 2 previous similar messages [ 5761.426981] Lustre: server umount lustre-MDT0000 complete [ 5764.948169] LustreError: 194777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689571 with bad export cookie 12131085259920117737 [ 5764.957228] LustreError: 194777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5764.959489] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5765.427263] Lustre: server umount lustre-MDT0001 complete [ 5788.750891] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 06:13:13 (1788689593) [ 5795.330370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5795.642894] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5795.695603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5796.007520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5800.828258] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5808.351981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.507320] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5808.539815] Lustre: 199211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5812.933823] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5819.675847] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5825.546648] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5832.364081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5832.598958] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5838.022883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5839.473749] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5847.874595] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5854.696511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5854.823059] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5861.391673] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5864.099976] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5868.423555] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5873.634487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5873.644341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5873.657358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5873.663926] Lustre: Skipped 2 previous similar messages [ 5876.743554] Lustre: server umount lustre-OST0000 complete [ 5879.463494] LustreError: 201065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5879.497170] LustreError: 201065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5895.161048] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5895.170264] LustreError: Skipped 1 previous similar message [ 5897.917794] Lustre: server umount lustre-OST0001 complete [ 5907.825275] Lustre: server umount lustre-MDT0000 complete [ 5910.990960] LustreError: 199203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689717 with bad export cookie 12131085259920119144 [ 5911.001894] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5911.007568] LustreError: 199203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5911.317653] Lustre: server umount lustre-MDT0001 complete [ 5936.139886] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 06:15:40 (1788689740) [ 5954.061555] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 5963.033317] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5999.627975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.911730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5999.944256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6000.050628] Lustre: lustre-MDT0000: new disk, initializing [ 6000.135040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6000.144379] Lustre: Skipped 3 previous similar messages [ 6000.162037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6004.958061] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6015.736748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6015.889810] Lustre: 207308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6015.969130] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6015.975387] Lustre: Skipped 1 previous similar message [ 6016.069833] Lustre: lustre-MDT0001: new disk, initializing [ 6016.241079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6016.257253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6020.627481] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6030.969104] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6037.257249] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6044.061966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6044.337375] Lustre: lustre-OST0000: new disk, initializing [ 6044.341875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6044.350874] Lustre: 209276:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6050.380885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6050.400365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6050.517186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6050.740738] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6061.754568] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6066.442997] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6066.695589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6072.423609] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6072.667048] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6076.386097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6076.392599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.412466] Lustre: Skipped 5 previous similar messages [ 6076.441317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6076.450242] Lustre: Skipped 8 previous similar messages [ 6089.184311] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6089.393497] Lustre: server umount lustre-OST0000 complete [ 6099.760064] Lustre: server umount lustre-MDT0000 complete [ 6101.988554] LustreError: 207320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6102.006373] LustreError: 207320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6103.511942] LustreError: 207301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689910 with bad export cookie 12131085259920120929 [ 6103.515916] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6103.529216] LustreError: 207301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6103.856808] Lustre: server umount lustre-MDT0001 complete [ 6112.369880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6117.251346] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6125.922698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6130.111655] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6138.548170] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6145.515145] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6152.535888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6152.821702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6152.829301] Lustre: Skipped 4 previous similar messages [ 6158.791140] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6166.725347] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6171.529152] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6171.796801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6176.377337] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6176.702881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6178.786179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6178.793290] LustreError: Skipped 1 previous similar message [ 6178.795619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6178.803542] Lustre: Skipped 3 previous similar messages [ 6178.807581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6178.810482] Lustre: Skipped 6 previous similar messages [ 6184.822894] Lustre: server umount lustre-OST0000 complete [ 6190.762201] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6195.307802] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6224.758339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6231.579814] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6232.313693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.330516] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6239.212408] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6243.244344] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6246.645922] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.667137] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6247.914062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6247.927452] LustreError: Skipped 1 previous similar message [ 6253.846723] Lustre: server umount lustre-OST0000 complete [ 6264.150726] Lustre: server umount lustre-MDT0000 complete [ 6265.315994] LustreError: 211493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6265.336200] LustreError: 211493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6268.053437] LustreError: 211477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788690074 with bad export cookie 12131085259920121811 [ 6268.082216] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6268.620425] Lustre: server umount lustre-MDT0001 complete [ 6277.686761] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 06:21:22 (1788690082) [ 6284.628841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6289.114670] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6297.174798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6301.898132] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6309.435634] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6315.358987] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6322.926993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6329.904448] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6337.266948] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6341.298981] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6341.459829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6345.637620] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6345.803245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6351.600718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6351.817548] Lustre: lustre-OST0001: new disk, initializing [ 6351.823318] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6351.839269] Lustre: 220829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6357.877909] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6358.138208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6358.158713] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6358.260210] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6368.442354] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6372.949468] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6373.146646] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6377.264945] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6377.425609] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6384.097723] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6384.099978] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6384.108952] LustreError: Skipped 3 previous similar messages [ 6384.114647] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6384.119375] Lustre: Skipped 6 previous similar messages [ 6384.129610] Lustre: Skipped 9 previous similar messages [ 6385.756086] Lustre: server umount lustre-OST0001 complete [ 6393.062093] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6393.346256] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6399.680461] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6400.028957] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6429.615750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6429.877254] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6429.888289] Lustre: Skipped 5 previous similar messages [ 6435.313122] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6435.351252] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6435.356127] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6435.388659] Lustre: Skipped 1 previous similar message [ 6436.699690] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6445.351686] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6447.530585] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6450.672827] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6450.682482] LustreError: Skipped 1 previous similar message [ 6456.008712] Lustre: server umount lustre-OST0001 complete [ 6487.008093] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6487.256438] Lustre: server umount lustre-OST0000 complete [ 6497.043183] Lustre: server umount lustre-MDT0000 complete [ 6501.133298] LustreError: 217441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788690307 with bad export cookie 12131085259920122777 [ 6501.152810] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6501.156608] LustreError: 217441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6507.608864] Lustre: server umount lustre-MDT0001 complete [ 6532.551896] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 06:25:36 (1788690336) [ 6539.482360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6539.859464] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6539.919348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6545.157450] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6553.793573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6553.864950] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6553.887295] Lustre: 226075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6558.454535] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6565.850343] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6571.615528] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6578.722594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6578.852867] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6585.085749] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6592.160454] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6601.109905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6601.209545] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6607.358369] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6614.131715] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6615.647684] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6619.367877] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6619.546436] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6622.977243] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6623.155457] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6627.776799] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c9f85270800.ost_server_uuid 50 [ 6629.018563] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c9f85270800.ost_server_uuid in IDLE state after 0 sec [ 6629.806684] Lustre: Permanently deactivating lustre-OST0001 [ 6629.821926] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6632.960144] Lustre: Permanently reactivating lustre-OST0001 [ 6641.126387] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6641.127491] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6641.127530] Lustre: Skipped 6 previous similar messages [ 6641.144750] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6641.167484] LustreError: Skipped 1 previous similar message [ 6641.169400] Lustre: Skipped 16 previous similar messages [ 6642.695765] Lustre: server umount lustre-OST0001 complete [ 6646.243946] LustreError: 229978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6646.265637] LustreError: 229978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 6652.264373] Lustre: server umount lustre-OST0000 complete [ 6662.447054] Lustre: server umount lustre-MDT0000 complete [ 6666.646780] LustreError: 226066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788690473 with bad export cookie 12131085259920124065 [ 6666.655902] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6666.668248] LustreError: 226066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6667.136351] Lustre: server umount lustre-MDT0001 complete [ 6690.416245] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 06:28:14 (1788690494) [ 6698.175064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6698.494773] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6698.545530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6698.553870] Lustre: Skipped 5 previous similar messages [ 6703.803991] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6712.933957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6713.031347] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6713.057651] Lustre: 232764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6718.693740] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6727.022490] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6733.451891] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6740.053519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6740.257761] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6740.272391] Lustre: Permanently deactivating lustre-OST0000 [ 6740.281379] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6746.163716] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6746.170848] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6746.183762] LustreError: 234989:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 6753.431654] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6760.139125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6760.228771] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6766.732185] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6766.931423] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6774.491470] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6776.773499] Lustre: Permanently reactivating lustre-OST0000 [ 6776.790869] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6776.797368] Lustre: Skipped 2 previous similar messages [ 6785.651435] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6785.652110] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 40 seconds. I think it's dead, and I am evicting it. exp 000000007e2dbce0, cur 1788690592 deadline 1788690582 last 1788690552 [ 6785.660102] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6785.661668] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6785.746861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6800.242631] Lustre: server umount lustre-OST0001 complete [ 6810.634503] Lustre: server umount lustre-OST0000 complete [ 6820.932937] Lustre: server umount lustre-MDT0000 complete [ 6824.392566] LustreError: 232756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788690630 with bad export cookie 12131085259920125710 [ 6824.402306] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6824.414900] LustreError: 232756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6824.721046] Lustre: server umount lustre-MDT0001 complete [ 6833.733461] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 6836.137459] Key type lgssc unregistered [ 6836.373218] LNet: 237645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6836.383857] LNetError: 237645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6836.401111] LNet: Removed LNI 192.168.203.160@tcp [ 6837.118892] Key type .llcrypt unregistered [ 6837.121157] Key type ._llcrypt unregistered [ 6859.417882] Key type ._llcrypt registered [ 6859.420554] Key type .llcrypt registered [ 6859.506135] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 6860.123596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6860.165318] alg: No test for adler32 (adler32-zlib) [ 6861.239098] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 6861.514700] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 6863.184215] Key type lgssc registered [ 6864.277744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6873.005345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6873.047519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6874.572434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6879.725062] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6884.839183] LustreError: 239107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6884.861852] LustreError: 239107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6889.548598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6889.666156] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6889.692225] Lustre: 239097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6889.707740] Lustre: Permanently deactivating lustre-MDT0001 [ 6889.720906] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6890.099438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6894.276893] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6899.220249] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6901.530284] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6907.195638] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6914.028970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6914.432803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6915.445918] LustreError: 240949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6915.469886] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 6919.528923] LustreError: 240950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6919.543500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6919.548431] LustreError: 240950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6920.453889] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6924.773725] LustreError: 240950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6924.794880] LustreError: 240950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6928.009818] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6929.899130] LustreError: 241035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6929.927130] LustreError: 241035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6935.406084] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6935.669657] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6941.205625] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6941.214957] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6942.552700] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6951.198565] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6954.462449] Lustre: Permanently reactivating lustre-MDT0001 [ 6954.477143] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6954.482116] Lustre: Skipped 2 previous similar messages [ 6959.303106] Lustre: lustre-MDT0001: Client f3dd6ee5-bff4-4465-8ad9-e846f313a97f (at 192.168.203.60@tcp) reconnecting [ 6960.265945] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6960.279141] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6960.289274] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6960.307332] LustreError: 242676:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6960.311636] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6960.326090] LustreError: 242676:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6960.326210] LustreError: 242676:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.160@tcp: cfg command failed: rc = -108 [ 6960.326232] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6960.326232] [ 6960.333896] LustreError: 239101:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6967.686221] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6969.609241] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6976.575105] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6976.760473] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6981.080859] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6981.267480] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6983.647988] Lustre: Permanently deactivating lustre-MDT0001 [ 6983.672713] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6983.678506] Lustre: Skipped 2 previous similar messages [ 6989.433760] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6997.489278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6997.489440] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6997.507791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6997.554868] Lustre: Skipped 2 previous similar messages [ 6998.372667] Lustre: server umount lustre-MDT0000 complete [ 7002.284431] LustreError: 239088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788690808 with bad export cookie 10139825971933443980 [ 7002.297231] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7002.574707] Lustre: server umount lustre-MDT0001 complete [ 7008.557509] Lustre: server umount lustre-OST0000 complete [ 7018.696717] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:33:43 (1788690823) [ 7028.745926] Lustre: server umount lustre-OST0001 complete [ 7035.685724] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 7045.143792] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7079.566304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7079.745113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7079.759658] Lustre: Skipped 2 previous similar messages [ 7079.792697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7079.906629] Lustre: lustre-MDT0000: new disk, initializing [ 7079.989355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7080.004232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7084.271792] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7094.379880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7094.492586] Lustre: 247368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7094.557636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7094.561896] Lustre: Skipped 1 previous similar message [ 7094.626657] Lustre: lustre-MDT0001: new disk, initializing [ 7094.702146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7094.733330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7094.739500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7098.811580] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7108.160504] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7113.798056] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7120.484252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7120.837640] Lustre: lustre-OST0000: new disk, initializing [ 7120.843111] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7120.852405] Lustre: 249336:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7120.956393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7126.617340] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7128.633076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7128.638402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7128.731319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7136.239395] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7138.735468] LustreError: 247374:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7148.194356] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7148.431629] Lustre: lustre-OST0001: new disk, initializing [ 7148.446896] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7148.452258] Lustre: 250453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7148.763748] LustreError: 247374:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7148.836791] LustreError: 250231:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7154.851691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7154.874037] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7154.970747] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7155.348761] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7158.936133] LustreError: 250231:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7164.718915] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7166.435576] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7169.720674] Lustre: lustre-OST0001: Not available for connect from 192.168.203.60@tcp (stopping) [ 7169.725195] Lustre: Skipped 3 previous similar messages [ 7170.531662] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7170.534430] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7170.551096] Lustre: Skipped 1 previous similar message [ 7174.673539] Lustre: server umount lustre-OST0001 complete [ 7174.831360] LustreError: 249318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7174.865140] LustreError: 249318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7190.498496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7190.499341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7190.505055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7190.508620] LustreError: Skipped 2 previous similar messages [ 7190.535934] Lustre: Skipped 3 previous similar messages [ 7191.016375] LustreError: 249319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7191.035464] LustreError: 249319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7195.856458] Lustre: server umount lustre-OST0000 complete [ 7201.251823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7201.257559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7201.268510] Lustre: Skipped 2 previous similar messages [ 7201.272729] Lustre: Skipped 3 previous similar messages [ 7205.464310] Lustre: server umount lustre-MDT0000 complete [ 7208.728131] LustreError: 247361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788691015 with bad export cookie 10139825971933447018 [ 7208.733134] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7208.741737] LustreError: 247361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7223.264127] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7223.593677] Lustre: server umount lustre-MDT0001 complete [ 7232.180424] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7234.463943] Key type lgssc unregistered [ 7234.741784] LNet: 252486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7234.752548] LNetError: 252486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7234.770212] LNet: Removed LNI 192.168.203.160@tcp [ 7235.365774] Key type .llcrypt unregistered [ 7235.367961] Key type ._llcrypt unregistered [ 7269.819986] Key type ._llcrypt registered [ 7269.824568] Key type .llcrypt registered [ 7269.993671] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7271.044143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7271.205740] alg: No test for adler32 (adler32-zlib) [ 7272.320269] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 7272.554132] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7274.248160] Key type lgssc registered [ 7275.477657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7282.498171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7282.517976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7283.777293] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7283.813736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7283.964786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7288.143935] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7294.408147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7294.476860] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7294.506418] Lustre: 254690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7294.676239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7298.945071] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7306.784751] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7312.549392] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7318.678426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7318.851046] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7319.047170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7324.500749] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7327.899404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7331.043402] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7384.039627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7384.054635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7384.092690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7384.552629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7384.557672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7385.923449] Lustre: server umount lustre-OST0000 complete [ 7390.201219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7402.092625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7402.520608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7404.585267] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7404.600229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7404.602286] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7408.106917] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7415.376511] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7455.715939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7455.726539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7455.728471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7455.735677] Lustre: Skipped 2 previous similar messages [ 7460.448799] Lustre: server umount lustre-OST0000 complete [ 7463.905955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7463.920499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.935992] Lustre: Skipped 1 previous similar message [ 7463.947938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7465.957257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7465.967834] Lustre: Skipped 1 previous similar message [ 7469.796874] Lustre: server umount lustre-MDT0000 complete [ 7474.078386] LustreError: 254682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788691280 with bad export cookie 4993357861415938736 [ 7474.079235] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7474.093430] LustreError: 254682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7474.674124] Lustre: server umount lustre-MDT0001 complete [ 7487.099498] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7489.503372] Key type lgssc unregistered [ 7489.741282] LNet: 260117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7489.751668] LNetError: 260117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7489.793590] LNet: Removed LNI 192.168.203.160@tcp [ 7490.620447] Key type .llcrypt unregistered [ 7490.622895] Key type ._llcrypt unregistered [ 7512.398767] Key type ._llcrypt registered [ 7512.402604] Key type .llcrypt registered [ 7512.507814] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7513.356082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7513.367803] alg: No test for adler32 (adler32-zlib) [ 7514.318554] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 7514.545354] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7516.192182] Key type lgssc registered [ 7517.148458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7523.549864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7523.575227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7524.999083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7529.256575] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7534.564508] LustreError: 261530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7534.579315] LustreError: 261530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7536.290273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7536.509622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7540.087900] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7546.749855] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7551.376682] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7557.272329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7557.603810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7563.065719] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7569.488514] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7569.905760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7592.418139] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7592.426234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7592.440737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7595.494313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7595.497790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7595.499475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7598.504500] Lustre: server umount lustre-OST0000 complete [ 7603.172815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7603.181902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7603.195944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7603.201771] Lustre: Skipped 1 previous similar message [ 7605.733573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7605.734475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7610.850798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7610.858110] Lustre: Skipped 2 previous similar messages [ 7616.992137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7617.230854] Lustre: server umount lustre-MDT0000 complete [ 7620.784285] LustreError: 265081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788691427 with bad export cookie 10119611125700384819 [ 7620.787486] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7620.796143] LustreError: 265081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7621.270170] Lustre: server umount lustre-MDT0001 complete [ 7631.104566] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7633.588336] Key type lgssc unregistered [ 7633.819868] LNet: 265833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7633.828464] LNetError: 265833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7633.838065] LNet: Removed LNI 192.168.203.160@tcp [ 7634.535455] Key type .llcrypt unregistered [ 7634.538675] Key type ._llcrypt unregistered [ 7649.598871] Key type ._llcrypt registered [ 7649.601207] Key type .llcrypt registered [ 7649.717649] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7650.457602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7650.620650] alg: No test for adler32 (adler32-zlib) [ 7651.631244] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 7651.910556] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7653.600315] Key type lgssc registered [ 7654.490165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7661.358689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7661.382809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7662.706962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7666.413070] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7673.315901] LustreError: 267052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7673.335612] LustreError: 267052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7675.043516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.427439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7680.406419] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7687.560691] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7693.137468] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7699.368595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7699.798171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7706.602839] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7707.963963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7714.537177] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7726.561909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7726.569609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7726.586299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7728.611698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7728.615954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7728.616055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7731.508745] Lustre: server umount lustre-OST0000 complete [ 7736.803232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7736.815500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7736.842436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7736.844871] Lustre: Skipped 1 previous similar message [ 7738.850503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7738.851918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7741.225752] Lustre: server umount lustre-MDT0000 complete [ 7743.972986] LustreError: 267047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7743.996524] LustreError: 267047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7744.944187] LustreError: 267032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788691551 with bad export cookie 252154352661365259 [ 7744.945743] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7744.954894] LustreError: 267032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7745.317162] Lustre: server umount lustre-MDT0001 complete [ 7758.934551] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7762.185606] Key type lgssc unregistered [ 7762.537626] LNet: 270730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7762.546751] LNetError: 270730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7762.564529] LNet: Removed LNI 192.168.203.160@tcp [ 7763.616450] Key type .llcrypt unregistered [ 7763.619660] Key type ._llcrypt unregistered [ 7787.719142] Key type ._llcrypt registered [ 7787.721320] Key type .llcrypt registered [ 7787.818896] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7788.825986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7788.844435] alg: No test for adler32 (adler32-zlib) [ 7789.994313] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 7790.242573] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7792.008679] Key type lgssc registered [ 7793.351979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7801.920120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7801.953628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7803.501417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7808.691087] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7814.121802] LustreError: 272142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7814.147659] LustreError: 272142:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7817.276820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7817.653404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7822.983318] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7830.332246] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7836.454582] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7844.256712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7844.605202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7848.712501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7853.416929] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7861.809902] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7888.869894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7888.880138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7888.883215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7888.897422] Lustre: Skipped 1 previous similar message [ 7893.996905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7894.009905] Lustre: Skipped 2 previous similar messages [ 7895.294904] Lustre: server umount lustre-OST0000 complete [ 7899.618107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7899.636167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7899.653523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7904.225907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7904.226257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7905.956339] Lustre: server umount lustre-MDT0000 complete [ 7909.346724] LustreError: 272143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7909.367154] LustreError: 272143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7910.224862] LustreError: 272122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788691716 with bad export cookie 17436368665694240294 [ 7910.228943] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7910.233684] LustreError: 272122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7910.722957] Lustre: server umount lustre-MDT0001 complete [ 7923.755195] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7926.785238] Key type lgssc unregistered [ 7927.145976] LNet: 276451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7927.163818] LNetError: 276451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7927.187259] LNet: Removed LNI 192.168.203.160@tcp [ 7928.681479] Key type .llcrypt unregistered [ 7928.694894] Key type ._llcrypt unregistered [ 7952.004259] Key type ._llcrypt registered [ 7952.011710] Key type .llcrypt registered [ 7952.262709] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7953.577964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7953.625490] alg: No test for adler32 (adler32-zlib) [ 7955.070570] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 7955.492942] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7957.344228] Key type lgssc registered [ 7958.699168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7967.142330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7967.163520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7968.716070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7973.559565] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7978.979145] LustreError: 277690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7978.996146] LustreError: 277690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7981.291142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7981.554185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7986.174966] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7994.132878] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8000.521571] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8007.738951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8008.096076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8013.884402] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8014.271388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8021.958214] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8036.845052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8036.847436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8036.858170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8036.874651] Lustre: Skipped 1 previous similar message [ 8041.953552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8041.965587] Lustre: Skipped 1 previous similar message [ 8042.425448] Lustre: server umount lustre-OST0000 complete [ 8047.078323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8047.102209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8047.102933] Lustre: Skipped 1 previous similar message [ 8047.134689] Lustre: Skipped 2 previous similar messages [ 8052.196299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8052.209079] Lustre: Skipped 1 previous similar message [ 8053.107486] Lustre: server umount lustre-MDT0000 complete [ 8057.217460] LustreError: 277656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788691863 with bad export cookie 17895965233960649480 [ 8057.234710] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8057.235958] LustreError: 277656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8057.777281] Lustre: server umount lustre-MDT0001 complete [ 8069.889892] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 8072.841933] Key type lgssc unregistered [ 8073.175598] LNet: 281389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8073.186516] LNetError: 281389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8073.223178] LNet: Removed LNI 192.168.203.160@tcp [ 8074.334581] Key type .llcrypt unregistered [ 8074.338751] Key type ._llcrypt unregistered [ 8086.743948] Key type ._llcrypt registered [ 8086.748484] Key type .llcrypt registered [ 8086.901520] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8103.884502] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 8121.673771] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 8123.229276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8123.260848] alg: No test for adler32 (adler32-zlib) [ 8124.302423] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 8124.506958] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 8126.224358] Key type lgssc registered [ 8127.427229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8163.364974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8163.419859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8164.815317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8164.859764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8164.954842] Lustre: lustre-MDT0000: new disk, initializing [ 8165.121482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8165.171215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8170.650989] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8181.858856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8182.019834] Lustre: 284949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8182.089579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8182.094859] Lustre: Skipped 1 previous similar message [ 8182.207412] Lustre: lustre-MDT0001: new disk, initializing [ 8182.317789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8182.348943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8182.365637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8186.765558] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8196.007798] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8202.527255] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8208.913971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8209.114344] Lustre: lustre-OST0000: new disk, initializing [ 8209.117399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8209.125411] Lustre: 286918:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8209.206326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8214.749836] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8218.171605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8218.181396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8218.285329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8223.898310] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8228.272514] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8228.514526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8233.920536] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8234.208832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8239.078357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8239.079424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8239.091767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8239.106317] Lustre: Skipped 1 previous similar message [ 8239.118238] Lustre: Skipped 1 previous similar message [ 8242.394255] Lustre: server umount lustre-OST0000 complete [ 8248.808337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8248.821387] LustreError: Skipped 1 previous similar message [ 8248.826790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8248.849383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8252.605594] Lustre: server umount lustre-MDT0000 complete [ 8257.033177] LustreError: 284943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788692063 with bad export cookie 18367203996768495566 [ 8257.046093] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8257.056148] LustreError: 284943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8257.951128] Lustre: server umount lustre-MDT0001 complete [ 8269.739728] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:54:33 (1788692073) [ 8277.715000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8278.106459] LustreError: 289334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8278.133870] LustreError: 289334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8278.235275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8283.336282] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8283.620573] LustreError: 289335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8287.716721] LustreError: 289334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8292.190654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8292.541103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8297.268646] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8305.616734] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8313.059852] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8320.754618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8321.110124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8328.033159] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8335.411329] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8354.284898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8354.288283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8354.297867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8354.297874] Lustre: Skipped 2 previous similar messages [ 8354.317358] Lustre: Skipped 2 previous similar messages [ 8355.674574] Lustre: server umount lustre-OST0000 complete [ 8364.518125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8364.524224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8364.541353] Lustre: Skipped 1 previous similar message [ 8364.554948] Lustre: Skipped 2 previous similar messages [ 8369.637707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8369.653996] Lustre: Skipped 1 previous similar message [ 8374.240255] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8374.430888] Lustre: server umount lustre-MDT0000 complete [ 8378.174855] LustreError: 289314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788692184 with bad export cookie 18367203996768496448 [ 8378.176218] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8378.184620] LustreError: 289314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8378.460965] Lustre: server umount lustre-MDT0001 complete [ 8390.041602] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 8393.428983] Key type lgssc unregistered [ 8393.684152] LNet: 292895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8393.692489] LNetError: 292895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8393.705478] LNet: Removed LNI 192.168.203.160@tcp [ 8394.609849] Key type .llcrypt unregistered [ 8394.614485] Key type ._llcrypt unregistered [ 8422.348970] Key type ._llcrypt registered [ 8422.351356] Key type .llcrypt registered [ 8422.450961] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 8438.604068] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 8440.029744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8440.125076] alg: No test for adler32 (adler32-zlib) [ 8441.223690] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 8441.479601] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 8443.184467] Key type lgssc registered [ 8444.404823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8494.858468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8494.880287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8496.220401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8496.285505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8496.443650] Lustre: lustre-MDT0000: new disk, initializing [ 8496.568376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8496.588460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8502.019176] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8512.946175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8513.055750] Lustre: 296717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8513.118318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8513.121381] Lustre: Skipped 1 previous similar message [ 8513.219451] Lustre: lustre-MDT0001: new disk, initializing [ 8513.326343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8513.378534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8513.384838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8517.258881] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8526.789898] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8532.761644] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8539.909630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8540.266873] Lustre: lustre-OST2710: new disk, initializing [ 8540.270442] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8540.282480] Lustre: 298686:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8540.370654] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8546.502640] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8548.913808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8548.925513] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8549.046674] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8555.662433] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8561.773447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8561.965572] Lustre: lustre-OST03e8: new disk, initializing [ 8561.969763] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8561.977650] Lustre: 299749:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8562.082232] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8568.284423] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8570.418765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8570.430603] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8570.508913] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8574.757540] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8581.483348] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8581.735828] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8585.704706] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8585.890730] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8590.940603] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8591.164399] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8595.876824] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8596.071131] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8605.666046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8605.680573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8605.708206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8606.692937] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8606.693586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8606.702593] Lustre: Skipped 1 previous similar message [ 8608.033068] Lustre: server umount lustre-MDT0000 complete [ 8611.708661] LustreError: 296711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788692418 with bad export cookie 17285169814977616318 [ 8611.709696] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8611.723045] LustreError: 296711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8611.811923] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8611.814730] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8611.819029] Lustre: Skipped 3 previous similar messages [ 8611.822963] Lustre: Skipped 1 previous similar message [ 8612.074739] Lustre: server umount lustre-MDT0001 complete [ 8622.074122] Lustre: server umount lustre-OST2710 complete [ 8625.434941] Lustre: server umount lustre-OST03e8 complete [ 8641.914319] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 8651.966110] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 8688.052352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8688.212308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8688.237762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8688.314172] Lustre: lustre-MDT0000: new disk, initializing [ 8688.379774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8688.394814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8692.484276] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8702.222223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8702.365262] Lustre: 305011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8702.404628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8702.408519] Lustre: Skipped 1 previous similar message [ 8702.479571] Lustre: lustre-MDT0001: new disk, initializing [ 8702.554300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8702.593075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8702.596639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8707.110084] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8715.981409] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8721.832673] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8728.820664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8729.085850] Lustre: lustre-OST0000: new disk, initializing [ 8729.094989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8729.106969] Lustre: 306978:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8729.200722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8734.800404] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8738.401478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8738.418108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8738.520338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8744.141892] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8748.165514] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8748.366653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8752.649850] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8752.850848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8759.264873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8759.267877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8759.271290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8759.271296] Lustre: Skipped 1 previous similar message [ 8759.314557] Lustre: Skipped 1 previous similar message [ 8760.713932] Lustre: server umount lustre-OST0000 complete [ 8764.386054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8764.397314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8764.412764] Lustre: Skipped 1 previous similar message [ 8764.437375] Lustre: Skipped 2 previous similar messages [ 8770.540966] Lustre: server umount lustre-MDT0000 complete [ 8774.172876] LustreError: 306977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788692580 with bad export cookie 17285169814977618012 [ 8774.176252] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8774.181464] LustreError: 306977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8774.588465] Lustre: server umount lustre-MDT0001 complete [ 8782.752205] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 07:03:07 (1788692587) [ 8784.023518] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8785.713236] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 07:03:10 (1788692590) [ 8788.839191] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 8810.813263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8811.005295] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8811.027169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8811.187523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8814.819071] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8822.117735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8822.233680] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8822.254845] Lustre: 310835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8825.997261] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8845.680888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8845.846443] Lustre: Denying initial registration attempt from nid 192.168.203.160@tcp, specified as failover [ 8845.853176] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8845.858639] LustreError: 312289:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 8845.869379] LustreError: 312289:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8845.875666] LustreError: 312289:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8845.940162] Lustre: server umount lustre-OST0000 complete [ 8845.943736] LustreError: 312289:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8853.475063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8853.484550] LustreError: Skipped 1 previous similar message [ 8853.491587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8853.507245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8853.509768] Lustre: Skipped 2 previous similar messages [ 8866.273162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8866.541351] Lustre: server umount lustre-MDT0000 complete [ 8869.721326] LustreError: 310825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788692676 with bad export cookie 17285169814977618894 [ 8869.729147] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8869.740848] LustreError: 310825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8870.009283] Lustre: server umount lustre-MDT0001 complete [ 8878.497379] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 8881.257996] Key type lgssc unregistered [ 8881.640668] LNet: 313409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8881.647626] LNetError: 313409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8881.665068] LNet: Removed LNI 192.168.203.160@tcp [ 8882.649349] Key type .llcrypt unregistered [ 8882.651808] Key type ._llcrypt unregistered [ 8893.791575] Key type ._llcrypt registered [ 8893.796368] Key type .llcrypt registered [ 8893.881452] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 8894.649317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8894.693740] alg: No test for adler32 (adler32-zlib) [ 8895.745486] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 8895.967920] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 8897.624165] Key type lgssc registered [ 8898.506935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8939.953238] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 8946.708676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8946.726394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8947.895495] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8947.913627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8948.054045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8951.799583] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8958.392139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8958.449624] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8958.472897] Lustre: 316164:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8958.700891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8962.095471] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8968.689094] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8973.481992] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8992.769415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8992.963867] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8993.140091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8998.730180] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9006.042249] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9009.638232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9009.648678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9009.664187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9010.146241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9010.159689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9010.184414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9010.188684] Lustre: Skipped 1 previous similar message [ 9014.022958] Lustre: server umount lustre-OST0000 complete [ 9020.387148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9020.400065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9020.403226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9020.406478] Lustre: Skipped 1 previous similar message [ 9023.931149] Lustre: server umount lustre-MDT0000 complete [ 9027.413859] LustreError: 316154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788692833 with bad export cookie 8826215004233942752 [ 9027.418398] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9027.431351] LustreError: 316154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9027.681566] Lustre: server umount lustre-MDT0001 complete [ 9037.201512] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 9039.282151] Key type lgssc unregistered [ 9039.549723] LNet: 319735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9039.564509] LNetError: 319735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9040.616595] LNet: Removed LNI 192.168.203.160@tcp [ 9041.333525] Key type .llcrypt unregistered [ 9041.335453] Key type ._llcrypt unregistered [ 9061.281626] Key type ._llcrypt registered [ 9061.284575] Key type .llcrypt registered [ 9061.362155] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 9061.930574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9061.947679] alg: No test for adler32 (adler32-zlib) [ 9062.938380] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 9063.152880] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 9064.824158] Key type lgssc registered [ 9065.704942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9072.258875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9072.297740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9073.743670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9077.121272] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9083.802523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9083.879753] LustreError: 321128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9083.906516] LustreError: 321128:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9084.142310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9087.866648] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9093.763248] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9098.622437] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9105.965267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9106.233576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9110.471529] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9116.147599] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9127.094265] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.60@tcp (stopping) [ 9128.933759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9128.942367] LustreError: 323211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9128.953508] LustreError: 323211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9128.969559] Lustre: server umount lustre-MDT0000 complete [ 9131.897560] LustreError: 321109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788692938 with bad export cookie 8080908300671712871 [ 9131.910485] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9132.188440] Lustre: server umount lustre-MDT0001 complete [ 9137.423839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9143.287401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9150.432376] Lustre: 320436:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788692940/real 1788692940] req@000000007af61fa9 x1875580413832576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788692956 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9150.450351] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9150.460900] Lustre: Skipped 2 previous similar messages [ 9152.416594] Lustre: 320437:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788692942/real 1788692942] req@0000000079ece236 x1875580413832704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788692958 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9154.528486] Lustre: 320438:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788692945/real 1788692945] req@000000006a429608 x1875580413832832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788692961 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9156.643705] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70252748f40b1a4b [ 9156.648969] Lustre: MGC192.168.203.160@tcp: Connection restored to 0@lo (at 0@lo) [ 9156.849959] LustreError: 322956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9156.935226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9156.999420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9157.924115] Lustre: 320439:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788692948/real 1788692948] req@000000004837ece0 x1875580413833088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788692964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9161.066954] LustreError: 324713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9161.091452] LustreError: 324713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9161.752412] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9163.104459] Lustre: 320436:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788692953/real 1788692953] req@000000009b812c0f x1875580413833216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788692969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9166.306156] LustreError: 322956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9166.323408] LustreError: 322956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9169.382446] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9169.886178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9170.357079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9174.260735] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9175.538751] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9181.433126] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9183.277276] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9188.888756] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9190.480600] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9196.005927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9196.013102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9196.044695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9196.051856] Lustre: Skipped 1 previous similar message [ 9199.089678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9199.102757] Lustre: Skipped 1 previous similar message [ 9201.892799] Lustre: server umount lustre-OST0000 complete [ 9206.251484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9206.265413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9206.285123] Lustre: Skipped 1 previous similar message [ 9206.288489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9211.928964] Lustre: server umount lustre-MDT0000 complete [ 9214.437554] LustreError: 324713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9214.449902] LustreError: 324713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9215.438784] LustreError: 326638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788693022 with bad export cookie 8080908300671720011 [ 9215.441492] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9215.463236] LustreError: 326638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9215.834190] Lustre: server umount lustre-MDT0001 complete [ 9229.037916] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 9232.407584] Key type lgssc unregistered [ 9232.761598] LNet: 327390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9232.778356] LNetError: 327390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9232.819270] LNet: Removed LNI 192.168.203.160@tcp [ 9234.109127] Key type .llcrypt unregistered [ 9234.110739] Key type ._llcrypt unregistered [ 9259.129364] Key type ._llcrypt registered [ 9259.133027] Key type .llcrypt registered [ 9259.259239] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 9260.277801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9260.342079] alg: No test for adler32 (adler32-zlib) [ 9261.400070] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 9261.669427] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 9263.384687] Key type lgssc registered [ 9264.503583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9271.456208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9271.475277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9272.971892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9276.271396] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9282.532546] LustreError: 328805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9282.559962] LustreError: 328805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9285.623100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9285.926523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9289.757251] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9297.632293] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9304.185566] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9308.645277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9308.649954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9308.667489] Lustre: Skipped 1 previous similar message [ 9308.680722] Lustre: Skipped 1 previous similar message [ 9310.753600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9310.760714] Lustre: Skipped 1 previous similar message [ 9313.375262] Lustre: server umount lustre-MDT0000 complete [ 9315.811664] LustreError: 328805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9315.831623] LustreError: 328805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9316.468627] LustreError: 328786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788693123 with bad export cookie 9217149122491064996 [ 9316.480157] LustreError: 328786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9316.482598] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9316.756024] Lustre: server umount lustre-MDT0001 complete [ 9330.278937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9330.565800] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9330.596872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9330.784976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9334.801562] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9342.493239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9342.599827] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9342.636525] Lustre: 331392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9342.850013] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9347.425925] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9355.325911] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9360.908300] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9370.668251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9370.939350] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9370.954399] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9370.966481] LustreError: 333280:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 9370.981093] LustreError: 333280:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9370.989506] LustreError: 333280:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9371.091902] Lustre: server umount lustre-OST0000 complete [ 9371.101243] LustreError: 333280:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9382.395916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9382.612647] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9382.768920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9388.791414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9390.263318] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9398.951424] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9407.133457] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9407.229366] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9407.284420] Lustre: lustre-OST0001: new disk, initializing [ 9407.287737] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9407.292635] Lustre: 334819:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9407.357727] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9412.547490] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9416.790583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9416.798876] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9416.910357] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9419.250937] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9424.573919] Lustre: server umount lustre-OST0001 complete [ 9427.426964] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9427.428093] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9427.440515] LustreError: Skipped 1 previous similar message [ 9427.447161] LustreError: 335417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9427.456900] Lustre: Skipped 1 previous similar message [ 9427.488671] LustreError: 335417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9429.985749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9429.995867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9430.015368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9432.548214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9432.550602] LustreError: 333748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9432.557479] Lustre: Skipped 2 previous similar messages [ 9432.579395] LustreError: 333748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9434.418526] Lustre: server umount lustre-OST0000 complete [ 9440.226097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9440.232855] LustreError: Skipped 1 previous similar message [ 9440.235942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9440.253207] Lustre: Skipped 1 previous similar message [ 9440.259566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9444.137025] Lustre: server umount lustre-MDT0000 complete [ 9447.907791] LustreError: 332423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9447.923977] LustreError: 332423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9447.957986] LustreError: 332132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788693254 with bad export cookie 9217149122491065549 [ 9447.964928] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9447.970264] LustreError: 332132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9448.351254] Lustre: server umount lustre-MDT0001 complete [ 9457.837727] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 9460.595794] Key type lgssc unregistered [ 9460.945966] LNet: 336777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9460.952530] LNetError: 336777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9460.981207] LNet: Removed LNI 192.168.203.160@tcp [ 9461.707838] Key type .llcrypt unregistered [ 9461.709786] Key type ._llcrypt unregistered [ 9528.886406] Key type ._llcrypt registered [ 9528.888908] Key type .llcrypt registered [ 9529.016766] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 9542.287817] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 9543.020874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9543.130549] alg: No test for adler32 (adler32-zlib) [ 9544.153530] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 9544.330306] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 9545.961365] Key type lgssc registered [ 9546.843728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9575.571686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9575.612021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9576.848917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9576.895245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9576.999874] Lustre: lustre-MDT0000: new disk, initializing [ 9577.103885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9577.134353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9581.520855] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9591.586595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9591.676227] Lustre: 341695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9591.730738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9591.734739] Lustre: Skipped 1 previous similar message [ 9591.824961] Lustre: lustre-MDT0001: new disk, initializing [ 9591.927898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9591.965426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9591.975075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9595.672561] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9605.449258] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9611.081767] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9617.926050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9618.187862] Lustre: lustre-OST0000: new disk, initializing [ 9618.192164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9618.199317] Lustre: 343663:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9618.300476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9624.272277] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9628.236349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9628.252063] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9628.351325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9633.124048] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9637.520541] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9637.695607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9641.976252] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9642.143576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9644.007310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9644.011318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9644.024916] LustreError: Skipped 1 previous similar message [ 9644.026427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9644.052278] Lustre: Skipped 1 previous similar message [ 9649.123908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9649.128142] Lustre: Skipped 1 previous similar message [ 9654.249833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9654.257778] Lustre: Skipped 1 previous similar message [ 9658.336444] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9658.552952] Lustre: server umount lustre-OST0000 complete [ 9663.459982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9663.472829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9663.488971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9663.491721] Lustre: Skipped 1 previous similar message [ 9664.487788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9668.453914] Lustre: server umount lustre-MDT0000 complete [ 9671.705234] LustreError: 341686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788693478 with bad export cookie 16174768155483262219 [ 9671.707904] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9671.715456] LustreError: 341686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9672.059361] Lustre: server umount lustre-MDT0001 complete [ 9679.650474] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 07:18:04 (1788693484) [ 9685.799514] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 07:18:10 (1788693490) [ 9694.190609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9694.551517] LustreError: 346416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9694.576206] LustreError: 346416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9694.623126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9698.700730] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9699.811548] LustreError: 346417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9704.929927] LustreError: 346416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9707.506936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9707.876888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9712.232153] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9720.194717] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9726.250104] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9732.355065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9732.667995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9737.827371] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9744.379661] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9751.042514] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9752.529421] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9754.075930] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9755.571354] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9757.073420] Lustre: Failing over lustre-MDT0000 [ 9757.288512] Lustre: server umount lustre-MDT0000 complete [ 9759.200995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9759.205205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9759.214035] LustreError: 346417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9765.348420] LustreError: 346411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9765.361594] LustreError: 346411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9774.439515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9774.565828] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9774.792549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9774.846218] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9776.805956] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9778.475584] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9780.201338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9780.227563] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9780.291750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9786.865577] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9788.330197] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9791.569080] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9793.375885] Lustre: Failing over lustre-MDT0000 [ 9793.786234] Lustre: server umount lustre-MDT0000 complete [ 9795.558722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9795.560590] LustreError: 346413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9795.569281] Lustre: Skipped 3 previous similar messages [ 9795.571778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9795.611040] LustreError: 346413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9811.152271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9811.241650] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9811.496486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9811.542334] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9812.636054] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9815.050251] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9816.550607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9816.554881] Lustre: Skipped 2 previous similar messages [ 9816.596810] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9816.634296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9822.558521] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9823.721422] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9831.906706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9831.906837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9831.918367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9831.918373] Lustre: Skipped 2 previous similar messages [ 9831.941615] Lustre: Skipped 2 previous similar messages [ 9834.226777] Lustre: server umount lustre-MDT0000 complete [ 9837.029770] LustreError: 346411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9837.053047] LustreError: 346411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 9837.378696] LustreError: 346399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788693643 with bad export cookie 16174768155483264928 [ 9837.383068] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9837.385269] LustreError: 346399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9837.653840] Lustre: server umount lustre-MDT0001 complete [ 9850.912752] Lustre: server umount lustre-OST0000 complete [ 9862.344621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9866.158590] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9872.237192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9875.408631] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9881.257371] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9886.065238] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9891.483206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9891.732819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9891.736388] Lustre: Skipped 2 previous similar messages [ 9896.288207] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9902.204831] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9909.175627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9917.413186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9917.420571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9917.433585] Lustre: Skipped 1 previous similar message [ 9917.438716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9917.447699] Lustre: Skipped 2 previous similar messages [ 9923.303158] Lustre: server umount lustre-OST0000 complete [ 9932.483665] Lustre: server umount lustre-MDT0000 complete [ 9934.817879] LustreError: 352752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9934.833623] LustreError: 352752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9935.801314] LustreError: 352733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788693742 with bad export cookie 16174768155483265677 [ 9935.808593] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9935.818752] LustreError: 352733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9936.097185] Lustre: server umount lustre-MDT0001 complete [ 9945.163179] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 9947.255626] Key type lgssc unregistered [ 9947.512319] LNet: 356306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9947.528141] LNetError: 356306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9947.545795] LNet: Removed LNI 192.168.203.160@tcp [ 9948.160702] Key type .llcrypt unregistered [ 9948.162639] Key type ._llcrypt unregistered [ 9968.630250] Key type ._llcrypt registered [ 9968.632855] Key type .llcrypt registered [ 9968.714942] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 9969.618673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9969.642517] alg: No test for adler32 (adler32-zlib) [ 9970.668179] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 9970.864937] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 9972.504173] Key type lgssc registered [ 9973.396725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9979.415727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9979.442287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9980.907918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9984.762513] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9991.137634] LustreError: 357813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9991.159676] LustreError: 357813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9991.795916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9992.019354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9994.876671] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10000.403370] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10004.479149] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10009.533061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10009.797592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10015.123468] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10021.292980] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10026.229333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10036.707274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10036.708691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10036.716882] Lustre: Skipped 2 previous similar messages [10036.726722] Lustre: Skipped 2 previous similar messages [10040.274393] Lustre: server umount lustre-MDT0000 complete [10041.829280] LustreError: 358958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10041.846808] LustreError: 358958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10042.991215] LustreError: 357794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788693849 with bad export cookie 6657290295017771242 [10042.997825] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10043.007392] LustreError: 357794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10043.212808] Lustre: server umount lustre-MDT0001 complete [10056.128392] Lustre: server umount lustre-OST0000 complete [10065.368505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10065.643561] LustreError: 361544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10065.696449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10068.351675] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10071.012379] LustreError: 361545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10074.245283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10074.530504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10077.716745] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10082.775709] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10086.990480] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10092.027413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10092.237586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10096.521780] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10101.998970] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10110.703813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10125.793858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10125.800841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10125.811079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10127.965227] Lustre: server umount lustre-MDT0000 complete [10130.564159] LustreError: 361524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788693937 with bad export cookie 6657290295017772348 [10130.568019] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10130.570508] LustreError: 361524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10130.802495] Lustre: server umount lustre-MDT0001 complete [10144.172484] Lustre: server umount lustre-OST0000 complete [10154.590075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10154.841399] LustreError: 365474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10154.877669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10157.323178] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10161.977068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10164.823674] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10168.959516] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10171.823312] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10175.890084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10179.803219] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10180.268622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10183.825269] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10187.619903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10187.620382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10187.629205] Lustre: Skipped 2 previous similar messages [10187.636750] Lustre: Skipped 4 previous similar messages [10192.766734] Lustre: server umount lustre-OST0000 complete [10197.987905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10197.991146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10197.992496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10198.021025] Lustre: Skipped 1 previous similar message [10201.299371] Lustre: server umount lustre-MDT0000 complete [10203.106063] LustreError: 365470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10203.117490] LustreError: 365470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10203.851943] LustreError: 365456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788694010 with bad export cookie 6657290295017773363 [10203.861493] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10204.089893] Lustre: server umount lustre-MDT0001 complete [10211.000848] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [10212.715745] Key type lgssc unregistered [10212.893775] LNet: 369003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10212.899395] LNetError: 369003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10212.911058] LNet: Removed LNI 192.168.203.160@tcp [10213.449893] Key type .llcrypt unregistered [10213.451941] Key type ._llcrypt unregistered [10231.260644] Key type ._llcrypt registered [10231.262017] Key type .llcrypt registered [10231.352315] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [10232.015129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10232.044101] alg: No test for adler32 (adler32-zlib) [10233.024127] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [10233.230596] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [10234.936556] Key type lgssc registered [10236.270366] Lustre: Echo OBD driver; http://www.lustre.org/ [10249.206474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10249.221279] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10249.227704] LustreError: 370419:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10249.243386] LustreError: 370419:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10249.248428] LustreError: 370419:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10249.253783] LustreError: 370419:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10249.259773] LustreError: 370419:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10260.250604] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10260.256280] LustreError: 370816:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10260.267647] LustreError: 370816:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10260.271505] LustreError: 370816:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10260.275587] LustreError: 370816:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10260.280649] LustreError: 370816:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10287.122263] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [10290.490215] Key type lgssc unregistered [10290.867841] LNet: 371819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10290.890738] LNetError: 371819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10290.922145] LNet: Removed LNI 192.168.203.160@tcp [10292.273550] Key type .llcrypt unregistered [10292.278561] Key type ._llcrypt unregistered [10313.238000] Key type ._llcrypt registered [10313.244344] Key type .llcrypt registered [10313.413339] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [10329.509166] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [10330.239034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10330.315666] alg: No test for adler32 (adler32-zlib) [10331.436854] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [10331.705497] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [10333.434198] Key type lgssc registered [10334.624175] Lustre: Echo OBD driver; http://www.lustre.org/ [10369.019332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10369.036899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10370.315363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10370.344670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10370.431474] Lustre: lustre-MDT0000: new disk, initializing [10370.523078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10370.551902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10374.889429] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10384.527720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10384.635380] Lustre: 374969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10384.695187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10384.700767] Lustre: Skipped 1 previous similar message [10384.795260] Lustre: lustre-MDT0001: new disk, initializing [10384.888834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10384.925865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10384.938408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10389.105962] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10398.360437] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10403.464497] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10409.543726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10409.791445] Lustre: lustre-OST0000: new disk, initializing [10409.802596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10409.808379] Lustre: 376938:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10409.879335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10415.245155] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10415.666339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10415.676676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10415.745271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10426.529723] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10432.407411] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10432.898548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10437.417034] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10437.622650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10441.187937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10441.209078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10441.211771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10441.231236] Lustre: Skipped 1 previous similar message [10446.028524] Lustre: server umount lustre-OST0000 complete [10451.427209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10451.427913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10451.436526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10451.436535] Lustre: Skipped 1 previous similar message [10451.443520] Lustre: Skipped 1 previous similar message [10456.150692] Lustre: server umount lustre-MDT0000 complete [10460.298719] LustreError: 374963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788694266 with bad export cookie 10073702719332537522 [10460.311276] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10460.690840] Lustre: server umount lustre-MDT0001 complete [10470.093481] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10472.590857] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 07:31:16 (1788694276) [10480.654136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10481.081686] LustreError: 379397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10481.093303] LustreError: 379397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10481.134602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10485.364333] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10486.241437] LustreError: 379398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10491.376840] LustreError: 379397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10494.770577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10495.028520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10500.466768] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10509.472995] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10515.238671] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10521.507936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10521.892183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10528.261277] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10536.525428] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10542.980127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10543.144791] Lustre: lustre-OST0001: new disk, initializing [10543.148367] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10543.155341] Lustre: 382233:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10543.233536] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10548.851284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10548.869745] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10549.032387] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10549.132824] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10558.137350] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10564.582619] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10564.583946] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10564.591715] LustreError: Skipped 1 previous similar message [10564.592303] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10564.592306] Lustre: Skipped 1 previous similar message [10564.643065] Lustre: Skipped 1 previous similar message [10567.870491] Lustre: server umount lustre-OST0001 complete [10569.711217] LustreError: 381220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10569.732733] LustreError: 381220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10574.822969] LustreError: 381221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10574.842934] LustreError: 381221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10585.064755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10585.068598] LustreError: 381222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10585.071246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10585.071255] Lustre: Skipped 2 previous similar messages [10585.086115] Lustre: Skipped 1 previous similar message [10585.128096] LustreError: 381222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10590.185365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10590.189374] Lustre: Skipped 2 previous similar messages [10598.880412] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10599.111391] Lustre: server umount lustre-OST0000 complete [10602.472802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10602.483086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10602.495077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10602.498295] Lustre: Skipped 2 previous similar messages [10608.667678] Lustre: server umount lustre-MDT0000 complete [10610.659608] LustreError: 379398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10610.673472] LustreError: 379398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10611.711995] LustreError: 379379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788694418 with bad export cookie 10073702719332538404 [10611.719552] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10611.723850] LustreError: 379379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10618.070727] Lustre: server umount lustre-MDT0001 complete [10629.017637] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [10631.022870] Key type lgssc unregistered [10631.205887] LNet: 384218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10631.212425] LNetError: 384218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10631.229453] LNet: Removed LNI 192.168.203.160@tcp [10631.911210] Key type .llcrypt unregistered [10631.913346] Key type ._llcrypt unregistered [10667.945204] Key type ._llcrypt registered [10667.947126] Key type .llcrypt registered [10667.994319] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [10668.658807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10668.673457] alg: No test for adler32 (adler32-zlib) [10669.663061] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [10669.828102] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [10671.449708] Key type lgssc registered [10672.276430] Lustre: Echo OBD driver; http://www.lustre.org/ [10676.784885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10676.801177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10677.951300] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10677.967280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10678.071341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10680.458904] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10685.467293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10685.539039] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10685.556571] Lustre: 386991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10685.738375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10687.920870] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10691.866325] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10695.522379] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10699.234113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10699.238850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10699.242826] Lustre: Skipped 1 previous similar message [10699.256630] Lustre: Skipped 1 previous similar message [10703.007997] Lustre: server umount lustre-MDT0000 complete [10705.118347] LustreError: 386984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788694511 with bad export cookie 17394057088420565944 [10705.126815] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10705.131513] LustreError: 386984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10705.336855] Lustre: server umount lustre-MDT0001 complete [10711.176740] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 07:35:16 (1788694516) [10715.382965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10715.661247] LustreError: 389504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10715.667779] LustreError: 389504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10715.692178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10717.611561] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10720.738269] LustreError: 389505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10722.321089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10722.499255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10724.594791] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10728.772773] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10732.423169] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10736.862129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10736.994632] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10737.114229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10740.798386] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10745.574555] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10751.721876] Lustre: Permanently deactivating lustre-OST0000 [10751.731700] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10752.400534] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10754.403244] LustreError: 392343:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10754.411141] LustreError: 392343:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10754.414797] LustreError: 392343:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10756.319554] LustreError: 392391:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10756.323692] LustreError: 392391:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10763.596812] Lustre: server umount lustre-OST0000 complete [10765.246983] LustreError: 392641:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10765.250708] LustreError: 392641:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10768.865278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10768.875223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10768.891486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10770.915844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10770.916547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10776.035341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10776.041286] Lustre: Skipped 2 previous similar messages [10777.071115] LustreError: 392738:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000397f8a05) refcount nonzero (1) after lock cleanup; forcing cleanup. [10777.105973] Lustre: server umount lustre-MDT0000 complete [10779.139277] LustreError: 391333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788694585 with bad export cookie 17394057088420566518 [10779.142912] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10779.146398] LustreError: 391333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10789.460162] Lustre: server umount lustre-MDT0001 complete [10794.431797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10797.056612] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10798.633889] LustreError: 393797:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10802.526476] Lustre: 394136:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.160@tcp [10803.843260] Lustre: server umount MGS complete [10809.704386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10809.929479] LustreError: 394860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10809.943143] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10809.947499] Lustre: Skipped 1 previous similar message [10809.980397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10811.932579] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10815.458271] LustreError: 394861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10816.188607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10816.348695] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10818.537298] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10822.691202] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10825.607906] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10826.703265] Lustre: Permanently reactivating lustre-OST0000 [10826.710979] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10826.718162] Lustre: Skipped 2 previous similar messages [10853.858103] LustreError: 396686:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10858.479150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10858.756180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10858.760187] Lustre: Skipped 1 previous similar message [10862.244726] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10866.574026] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10878.433470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10878.440855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10878.448326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10879.457043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10880.687875] Lustre: server umount lustre-OST0000 complete [10883.040997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10883.047235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10883.057885] Lustre: Skipped 1 previous similar message [10889.697479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10889.703867] Lustre: Skipped 6 previous similar messages [10896.864110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10896.961855] Lustre: server umount lustre-MDT0000 complete [10899.354432] LustreError: 398460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788694705 with bad export cookie 17394057088420567561 [10899.360485] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10899.362313] LustreError: 398460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10899.542768] Lustre: server umount lustre-MDT0001 complete [10906.321672] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [10908.109909] Key type lgssc unregistered [10908.302913] LNet: 399412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10908.309430] LNetError: 399412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10908.326841] LNet: Removed LNI 192.168.203.160@tcp [10908.932118] Key type .llcrypt unregistered [10908.935389] Key type ._llcrypt unregistered [10921.031072] Key type ._llcrypt registered [10921.033755] Key type .llcrypt registered [10921.133749] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [10932.122239] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [10932.688969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10932.725307] alg: No test for adler32 (adler32-zlib) [10933.699284] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [10933.840585] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [10935.465046] Key type lgssc registered [10936.288722] Lustre: Echo OBD driver; http://www.lustre.org/ [10957.739308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10957.762405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10958.885435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10958.911423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10958.961497] Lustre: lustre-MDT0000: new disk, initializing [10959.042416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10959.054199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10961.376419] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10967.470755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10967.546617] Lustre: 402542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10967.581412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10967.584562] Lustre: Skipped 1 previous similar message [10967.635788] Lustre: lustre-MDT0001: new disk, initializing [10967.699756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10967.729946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10967.736627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10970.385742] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10975.732308] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10978.697676] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10982.606795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10982.790907] Lustre: lustre-OST0000: new disk, initializing [10982.793890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10982.798707] Lustre: 404510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10982.870156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10986.082642] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10991.463535] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10992.661096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10992.666795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10992.746722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10994.004446] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10994.118691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10996.396401] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10996.524244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10997.731468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10997.746594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10997.748264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10999.804706] Lustre: server umount lustre-OST0000 complete [11002.850315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11002.851108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11002.862241] Lustre: Skipped 2 previous similar messages [11002.865349] Lustre: Skipped 1 previous similar message [11007.969496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11007.974411] Lustre: Skipped 1 previous similar message [11008.177079] Lustre: server umount lustre-MDT0000 complete [11010.034391] LustreError: 403335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788694816 with bad export cookie 554667143182892017 [11010.039052] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11010.041660] LustreError: 403335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11010.211471] Lustre: server umount lustre-MDT0001 complete [11014.576896] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11015.507035] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 07:40:20 (1788694820) [11018.893642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11019.215874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11021.044281] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11024.356451] LustreError: 406964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11024.365432] LustreError: 406964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11025.138730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11025.343278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11027.235678] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11030.924496] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11033.727585] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11036.836106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11037.006311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11039.771327] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11043.412397] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11044.403799] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11046.237871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11054.561379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11054.565748] LustreError: Skipped 1 previous similar message [11054.567831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11054.574423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11055.585881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11059.756815] Lustre: server umount lustre-OST0000 complete [11065.825752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11065.828115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11065.830827] Lustre: Skipped 2 previous similar messages [11065.835782] Lustre: Skipped 2 previous similar messages [11067.854138] Lustre: server umount lustre-MDT0000 complete [11069.764719] LustreError: 406943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788694876 with bad export cookie 554667143182892892 [11069.765170] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11069.772585] LustreError: 406943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11069.954158] Lustre: server umount lustre-MDT0001 complete [11074.955840] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [11076.353823] Key type lgssc unregistered [11076.505422] LNet: 410683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11076.510594] LNetError: 410683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11076.518700] LNet: Removed LNI 192.168.203.160@tcp [11076.916806] Key type .llcrypt unregistered [11076.918258] Key type ._llcrypt unregistered [11088.062800] Key type ._llcrypt registered [11088.064706] Key type .llcrypt registered [11088.120465] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [11088.282402] Key type .llcrypt unregistered [11088.284066] Key type ._llcrypt unregistered [11098.157166] Key type ._llcrypt registered [11098.158772] Key type .llcrypt registered [11098.215140] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [11098.607690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11098.616916] alg: No test for adler32 (adler32-zlib) [11099.511114] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [11099.628094] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [11101.216211] Key type lgssc registered [11101.688815] Lustre: Echo OBD driver; http://www.lustre.org/ [11105.121074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11105.129973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11106.328838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11108.336842] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11112.964384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11113.153451] LustreError: 412878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11113.163377] LustreError: 412878:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11113.171158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11116.264881] LustreError: 412897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11116.272821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11116.284761] LustreError: 412897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11116.428436] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11120.772244] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11121.634820] LustreError: 412878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11121.641642] LustreError: 412878:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11124.570165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11124.760888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11126.974031] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11136.994226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11136.999075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11137.006234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11140.068941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11140.072099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11140.077392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11141.941924] Lustre: server umount lustre-OST0000 complete [11145.186046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11145.189992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11145.199621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11145.206181] Lustre: Skipped 1 previous similar message [11145.209351] Lustre: Skipped 1 previous similar message [11150.126974] Lustre: server umount lustre-MDT0000 complete [11150.305560] LustreError: 412897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11150.319473] LustreError: 412897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11151.941149] LustreError: 412859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788694958 with bad export cookie 5543318315414539298 [11151.942456] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11151.948827] LustreError: 412859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11152.097146] Lustre: server umount lustre-MDT0001 complete [11157.429488] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [11158.955613] Key type lgssc unregistered [11159.138126] LNet: 416024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11159.142042] LNetError: 416024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11159.153163] LNet: Removed LNI 192.168.203.160@tcp [11159.622559] Key type .llcrypt unregistered [11159.624769] Key type ._llcrypt unregistered [11171.582413] Key type ._llcrypt registered [11171.583653] Key type .llcrypt registered [11171.628703] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [11171.968258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11171.975718] alg: No test for adler32 (adler32-zlib) [11172.882497] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [11172.990759] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [11174.576151] Key type lgssc registered [11175.140378] Lustre: Echo OBD driver; http://www.lustre.org/ [11178.460735] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11178.471667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11193.952391] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11199.072371] LustreError: 417387:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.160@tcp: failed processing log, type 4: rc = -110 [11225.696158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11228.378686] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11231.893455] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11235.288860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11235.564595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11235.605130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11237.706896] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11240.675807] LustreError: 418425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11240.683484] LustreError: 418425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11242.175272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11242.377845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11244.307835] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11248.172796] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11250.669471] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11256.288862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11256.292666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11256.299234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11257.826604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11257.828269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11257.831801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11259.611568] Lustre: server umount lustre-OST0000 complete [11262.944942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11262.946433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11262.950623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11262.955718] Lustre: Skipped 1 previous similar message [11262.958401] Lustre: Skipped 1 previous similar message [11267.440071] Lustre: server umount lustre-MDT0000 complete [11268.065397] LustreError: 418423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11268.072807] LustreError: 418423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11268.998515] LustreError: 417393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788695075 with bad export cookie 1450062827422528222 [11269.000966] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11269.003724] LustreError: 417393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11269.163474] Lustre: server umount lustre-MDT0001 complete [11273.949631] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [11275.243177] Key type lgssc unregistered [11275.384656] LNet: 420950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11275.388867] LNetError: 420950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11275.396638] LNet: Removed LNI 192.168.203.160@tcp [11275.801407] Key type .llcrypt unregistered [11275.803040] Key type ._llcrypt unregistered [11286.459587] Key type ._llcrypt registered [11286.461394] Key type .llcrypt registered [11286.502887] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [11286.964977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11286.983675] alg: No test for adler32 (adler32-zlib) [11287.879239] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [11287.997742] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [11289.592442] Key type lgssc registered [11290.095360] Lustre: Echo OBD driver; http://www.lustre.org/ [11293.256920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11293.265101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11294.460732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11296.101992] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11299.398837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11299.538440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11301.088429] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11303.826763] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11305.981131] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11308.622990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11308.753331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11310.817442] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11313.499206] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11320.044515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11325.409686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11325.411051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11325.411831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11325.421637] Lustre: Skipped 1 previous similar message [11329.191328] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.60@tcp (stopping) [11329.194750] Lustre: Skipped 2 previous similar messages [11330.529726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11330.532762] Lustre: Skipped 2 previous similar messages [11331.232049] Lustre: server umount lustre-MDT0000 complete [11332.191708] LustreError: 422320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.60@tcp arrived at 1788695138 with bad export cookie 8738011258096823989 [11332.199884] LustreError: 422320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11332.922371] LustreError: 422321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788695139 with bad export cookie 8738011258096823338 [11332.927346] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11335.649587] LustreError: 422341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11335.650411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11335.650590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11335.650595] Lustre: Skipped 1 previous similar message [11335.651202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11335.659036] LustreError: 422341:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11337.696868] LustreError: 422359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11337.704123] LustreError: 422359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11339.047944] Lustre: server umount lustre-OST0000 complete [11341.858801] Lustre: server umount lustre-MDT0001 complete [11346.323627] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [11347.557323] Key type lgssc unregistered [11347.696574] LNet: 426041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11347.700957] LNetError: 426041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11347.712718] LNet: Removed LNI 192.168.203.160@tcp [11348.079234] Key type .llcrypt unregistered [11348.080931] Key type ._llcrypt unregistered [11359.736681] Key type ._llcrypt registered [11359.739066] Key type .llcrypt registered [11359.789380] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [11360.150744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11360.197069] alg: No test for adler32 (adler32-zlib) [11361.096456] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [11361.204526] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [11362.816202] Key type lgssc registered [11363.309115] Lustre: Echo OBD driver; http://www.lustre.org/ [11366.872599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11366.883184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11368.091173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11369.929864] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11373.662090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11373.840748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11375.543899] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11378.773573] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11381.214654] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11384.203023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11384.379529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11386.786319] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11389.820153] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11392.566204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11398.624907] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11398.631710] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11398.638564] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11402.392733] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.60@tcp (stopping) [11402.723727] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11404.183308] Lustre: server umount lustre-MDT0001 complete [11407.841235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11407.842707] LustreError: 428171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11407.846661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11407.853956] LustreError: 428171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11407.860476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11407.867602] Lustre: Skipped 2 previous similar messages [11412.447980] Lustre: server umount lustre-OST0000 complete [11412.961646] LustreError: 427430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11414.140572] Lustre: server umount lustre-MDT0000 complete [11419.636313] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [11420.819367] Key type lgssc unregistered [11420.962600] LNet: 431109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11420.966279] LNetError: 431109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11420.976368] LNet: Removed LNI 192.168.203.160@tcp [11421.326844] Key type .llcrypt unregistered [11421.328581] Key type ._llcrypt unregistered [11430.581115] Key type ._llcrypt registered [11430.583325] Key type .llcrypt registered [11430.627310] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [11437.289791] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [11437.766462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11437.775767] alg: No test for adler32 (adler32-zlib) [11438.653662] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [11438.763189] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [11440.360163] Key type lgssc registered [11440.766100] Lustre: Echo OBD driver; http://www.lustre.org/ [11454.769372] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [11460.421145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11460.430153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11461.517331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11461.529265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11461.570574] Lustre: lustre-MDT0000: new disk, initializing [11461.594801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11461.602351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11463.218545] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11469.801246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11469.913921] Lustre: lustre-OST0000: new disk, initializing [11469.916656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11469.919652] Lustre: Skipped 1 previous similar message [11469.922845] Lustre: 436107:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11469.959678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11470.212661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11470.216491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11470.230144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11472.271491] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11476.579077] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11482.942080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11482.997447] Lustre: 435089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11483.034464] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11483.077764] Lustre: lustre-MDT0001: new disk, initializing [11483.127937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11483.148513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11483.152893] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11484.740933] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11488.793210] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11488.881866] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11490.398264] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11491.511276] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [11495.584102] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.60@tcp (stopping) [11496.420090] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11496.420802] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11496.423596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11496.427204] Lustre: Skipped 1 previous similar message [11496.431034] Lustre: Skipped 1 previous similar message [11499.350776] Lustre: server umount lustre-MDT0001 complete [11506.658114] LustreError: 435097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11506.663056] LustreError: 435097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11511.778557] LustreError: 438409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11511.785527] LustreError: 438409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11516.385154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11516.388831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11516.394272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11516.897750] LustreError: 435101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11522.017369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11522.017551] LustreError: 435102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11526.112189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11526.174873] Lustre: server umount lustre-OST0000 complete [11527.137940] LustreError: 435101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11533.705876] Lustre: server umount lustre-MDT0000 complete [11538.801684] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [11539.927623] Key type lgssc unregistered [11540.067431] LNet: 439861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11540.071801] LNetError: 439861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11540.081328] LNet: Removed LNI 192.168.203.160@tcp [11540.403937] Key type .llcrypt unregistered [11540.405412] Key type ._llcrypt unregistered [11554.194564] Key type ._llcrypt registered [11554.196029] Key type .llcrypt registered [11554.243624] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [11554.359440] Key type .llcrypt unregistered [11554.360968] Key type ._llcrypt unregistered [11561.425323] Key type ._llcrypt registered [11561.426700] Key type .llcrypt registered [11561.469668] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [11561.899667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11561.920379] alg: No test for adler32 (adler32-zlib) [11562.782239] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [11562.875760] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [11564.464175] Key type lgssc registered [11564.849074] Lustre: Echo OBD driver; http://www.lustre.org/ [11574.679326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11574.685822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11575.772792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11575.786210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11575.826132] Lustre: lustre-MDT0000: new disk, initializing [11575.884080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11575.893302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11577.562963] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11582.173981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11582.213961] Lustre: 443201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11582.239472] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11582.241766] Lustre: Skipped 1 previous similar message [11582.280447] Lustre: lustre-MDT0001: new disk, initializing [11582.318275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11582.332059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11582.335642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11583.821408] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11587.559423] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11589.403179] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11591.692387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11591.809392] Lustre: lustre-OST0000: new disk, initializing [11591.811705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11591.815879] Lustre: 445167:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11591.845046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11592.255567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11592.259893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11592.288127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11593.790127] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11597.556756] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11602.404924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11602.405861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11602.409137] Lustre: Skipped 2 previous similar messages [11602.411626] Lustre: Skipped 1 previous similar message [11606.121061] Lustre: server umount lustre-MDT0000 complete [11607.522504] LustreError: 441734:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ed4361e5 x1875583035273344/t0(0) o38->lustre-MDT0000-lwp-MDT0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11607.523260] LustreError: 443192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788695414 with bad export cookie 15467761713200890553 [11607.526894] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11607.538133] LustreError: 443192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11607.639907] Lustre: server umount lustre-MDT0001 complete [11619.171495] Lustre: server umount lustre-OST0000 complete [11622.728371] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 07:50:28 (1788695428) [11625.344929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11625.500899] LustreError: 447196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11625.508395] LustreError: 447196:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11625.528910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11626.972283] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11629.832941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11629.951750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11631.329708] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11633.818392] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11635.684522] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11650.979585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11651.058032] Lustre: Found index 0 for lustre-OST0000, updating log [11651.121736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11653.119137] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11655.608823] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11671.590142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11680.736870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11680.739782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11680.743990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11681.762422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11681.765349] Lustre: Skipped 2 previous similar messages [11683.604596] Lustre: server umount lustre-OST0000 complete [11696.608890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11696.613432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11696.619283] Lustre: Skipped 1 previous similar message [11696.622157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11702.426855] Lustre: server umount lustre-MDT0000 complete [11703.264948] LustreError: 447196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11703.270011] LustreError: 447196:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11703.697660] LustreError: 449036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788695510 with bad export cookie 15467761713200891932 [11703.702102] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11703.702281] LustreError: 449036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11703.829497] Lustre: server umount lustre-MDT0001 complete [11706.675346] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 07:51:52 (1788695512) [11711.392212] LNet: 1 local NIs in recovery (showing 1): 192.168.203.160@tcp [12287.030689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12287.172196] LustreError: 453633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12287.193914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12288.620254] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12291.509535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12291.629880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12292.974395] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12295.295376] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12297.057223] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12312.290946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12312.380071] Lustre: Found index 0 for lustre-OST0000, updating log [12312.484366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12314.329051] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12314.534749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [12316.712672] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12318.434239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12318.439572] Lustre: Skipped 1 previous similar message [12318.441939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12318.444021] Lustre: Skipped 2 previous similar messages [12319.457712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12319.460282] Lustre: Skipped 1 previous similar message [12323.464509] Lustre: server umount lustre-OST0000 complete [12324.833094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12324.834144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12324.837039] Lustre: Skipped 1 previous similar message [12326.881114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12326.883081] Lustre: Skipped 4 previous similar messages [12330.772639] Lustre: server umount lustre-MDT0000 complete [12330.976918] LustreError: 456284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12330.981039] LustreError: 456284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12331.930067] LustreError: 453613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788696138 with bad export cookie 15467761713200893003 [12331.932761] LustreError: 453613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12331.933199] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12332.033809] Lustre: server umount lustre-MDT0001 complete [12338.080225] LNet: 1 local NIs in recovery (showing 1): 192.168.203.160@tcp [12349.181283] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [12352.357180] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12364.479754] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 08:02:49 (1788696169) [12370.118253] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [12371.148102] Key type lgssc unregistered [12375.392363] LNet: 460285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12375.395570] LNetError: 460285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12445.030433] LNet: Removed LNI 192.168.203.160@tcp [12445.345429] Key type .llcrypt unregistered [12445.346742] Key type ._llcrypt unregistered [12451.901281] Key type ._llcrypt registered [12451.902612] Key type .llcrypt registered [12451.939649] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12452.276895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12452.297174] alg: No test for adler32 (adler32-zlib) [12453.157172] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [12453.241647] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [12454.824130] Key type lgssc registered [12455.161316] Lustre: Echo OBD driver; http://www.lustre.org/ [12456.583031] LNet: Added LNI 192.168.203.160@tcp100 [8/256/0/180] [12456.878681] LNet: Added LNI 192.168.203.160@tcp101 [8/256/0/180] [12457.157909] LNet: Added LNI 192.168.203.160@tcp102 [8/256/0/180] [12457.445237] LNet: Added LNI 192.168.203.160@tcp103 [8/256/0/180] [12457.723123] LNet: Added LNI 192.168.203.160@tcp104 [8/256/0/180] [12458.000309] LNet: Added LNI 192.168.203.160@tcp105 [8/256/0/180] [12458.276866] LNet: Added LNI 192.168.203.160@tcp106 [8/256/0/180] [12458.542889] LNet: Added LNI 192.168.203.160@tcp107 [8/256/0/180] [12458.826514] LNet: Added LNI 192.168.203.160@tcp108 [8/256/0/180] [12459.117756] LNet: Added LNI 192.168.203.160@tcp109 [8/256/0/180] [12459.406418] LNet: Added LNI 192.168.203.160@tcp110 [8/256/0/180] [12459.688810] LNet: Added LNI 192.168.203.160@tcp111 [8/256/0/180] [12459.975610] LNet: Added LNI 192.168.203.160@tcp112 [8/256/0/180] [12460.259485] LNet: Added LNI 192.168.203.160@tcp113 [8/256/0/180] [12460.542955] LNet: Added LNI 192.168.203.160@tcp114 [8/256/0/180] [12460.833791] LNet: Added LNI 192.168.203.160@tcp115 [8/256/0/180] [12461.106342] LNet: Added LNI 192.168.203.160@tcp116 [8/256/0/180] [12461.383296] LNet: Added LNI 192.168.203.160@tcp117 [8/256/0/180] [12461.662347] LNet: Added LNI 192.168.203.160@tcp118 [8/256/0/180] [12461.941058] LNet: Added LNI 192.168.203.160@tcp119 [8/256/0/180] [12462.225607] LNet: Added LNI 192.168.203.160@tcp120 [8/256/0/180] [12462.511044] LNet: Added LNI 192.168.203.160@tcp121 [8/256/0/180] [12462.790770] LNet: Added LNI 192.168.203.160@tcp122 [8/256/0/180] [12463.061384] LNet: Added LNI 192.168.203.160@tcp123 [8/256/0/180] [12463.338394] LNet: Added LNI 192.168.203.160@tcp124 [8/256/0/180] [12463.616282] LNet: Added LNI 192.168.203.160@tcp125 [8/256/0/180] [12463.888845] LNet: Added LNI 192.168.203.160@tcp126 [8/256/0/180] [12464.159738] LNet: Added LNI 192.168.203.160@tcp127 [8/256/0/180] [12464.436401] LNet: Added LNI 192.168.203.160@tcp128 [8/256/0/180] [12464.703405] LNet: Added LNI 192.168.203.160@tcp129 [8/256/0/180] [12464.980087] LNet: Added LNI 192.168.203.160@tcp130 [8/256/0/180] [12465.248150] LNet: Added LNI 192.168.203.160@tcp131 [8/256/0/180] [12465.524625] LNet: Added LNI 192.168.203.160@tcp132 [8/256/0/180] [12465.799548] LNet: Added LNI 192.168.203.160@tcp133 [8/256/0/180] [12466.078885] LNet: Added LNI 192.168.203.160@tcp134 [8/256/0/180] [12466.362551] LNet: Added LNI 192.168.203.160@tcp135 [8/256/0/180] [12468.478618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12468.485350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12469.519608] Lustre: 463277:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.160@tcp: can't add NID 192.168.203.160@tcp130: rc = -75 [12469.546832] LNetError: 460743:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.160 [12469.554165] LNetError: Protocol error connecting to 192.168.203.160@tcp115 at host 192.168.203.160:988: is it running a compatible version of Lustre? [12469.557324] LNetError: Skipped 4 previous similar messages [12469.589180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12469.598443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12469.631627] Lustre: lustre-MDT0000: new disk, initializing [12469.651889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12469.657332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12469.664085] LNet: 6 local NIs in recovery (showing 6): 192.168.203.160@tcp124, 192.168.203.160@tcp122, 192.168.203.160@tcp126, 192.168.203.160@tcp127, 192.168.203.160@tcp128, 192.168.203.160@tcp129 [12470.562293] LNetError: 463298:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.160 [12470.565241] LNetError: 463298:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [12470.568047] LNetError: Protocol error connecting to 192.168.203.160@tcp100 at host 192.168.203.160:988: is it running a compatible version of Lustre? [12470.573108] LNetError: Skipped 3 previous similar messages [12470.947054] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12472.738963] LNetError: 460742:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.160 [12472.742314] LNetError: 460742:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [12472.745279] LNetError: Protocol error connecting to 192.168.203.160@tcp122 at host 192.168.203.160:988: is it running a compatible version of Lustre? [12472.748883] LNetError: Skipped 7 previous similar messages [12474.815541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12474.859121] Lustre: 463295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12474.880854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12474.882552] Lustre: Skipped 1 previous similar message [12474.915061] Lustre: lustre-MDT0001: new disk, initializing [12474.925755] Lustre: 464097:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.160@tcp: can't add NID 192.168.203.160@tcp130: rc = -75 [12474.928086] Lustre: 464097:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12474.964145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12474.974234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12474.977079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12476.255594] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12476.834416] LNetError: 463297:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.160 [12476.838084] LNetError: 463297:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [12476.840583] LNetError: Protocol error connecting to 192.168.203.160@tcp122 at host 192.168.203.160:988: is it running a compatible version of Lustre? [12478.257580] Lustre: 464522:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.160@tcp: can't add NID 192.168.203.160@tcp130: rc = -75 [12478.260145] Lustre: 464522:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12479.842341] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12481.573722] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12483.727160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12483.831745] Lustre: lustre-OST0000: new disk, initializing [12483.833553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12483.836091] Lustre: 465270:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12483.864648] Lustre: 465275:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.160@tcp: can't add NID 192.168.203.160@tcp130: rc = -75 [12483.867554] Lustre: 465275:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [12483.883341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12485.026405] LNetError: 463303:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.160 [12485.029230] LNetError: Protocol error connecting to 192.168.203.160@tcp122 at host 192.168.203.160:988: is it running a compatible version of Lustre? [12485.642298] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12487.420885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12487.423957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12487.465464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12489.124647] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12492.772088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12492.772278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12492.773037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12492.775705] Lustre: Skipped 1 previous similar message [12493.392249] Lustre: server umount lustre-OST0000 complete [12495.328806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12495.331400] LustreError: Skipped 1 previous similar message [12495.332719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12495.336967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12495.338737] Lustre: Skipped 1 previous similar message [12497.889322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12497.889681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12500.385050] LNetError: 463311:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.160 [12500.388545] LNetError: Protocol error connecting to 192.168.203.160@tcp100 at host 192.168.203.160:988: is it running a compatible version of Lustre? [12500.760795] Lustre: server umount lustre-MDT0000 complete [12501.921070] LustreError: 463287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788696308 with bad export cookie 7035191943436154540 [12501.924283] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12501.924520] LustreError: 463287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12502.032827] Lustre: server umount lustre-MDT0001 complete [12505.425761] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [12506.437983] Key type lgssc unregistered [12506.606301] LNet: 467179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12506.609843] LNetError: 467179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12506.612349] LNet: Removed LNI 192.168.203.160@tcp [12507.616209] LNet: Removed LNI 192.168.203.160@tcp100 [12507.617739] LNetError: 467179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12507.619925] LNetError: 467179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12507.622358] LNet: Removed LNI 192.168.203.160@tcp101 [12508.640170] LNet: Removed LNI 192.168.203.160@tcp102 [12508.641728] LNetError: 467179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12508.644081] LNetError: 467179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12508.646615] LNet: Removed LNI 192.168.203.160@tcp103 [12509.664135] LNet: Removed LNI 192.168.203.160@tcp104 [12510.689382] LNet: Removed LNI 192.168.203.160@tcp105 [12510.691229] LNetError: 467179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12510.693425] LNetError: 467179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12510.696724] LNet: Removed LNI 192.168.203.160@tcp106 [12511.712557] LNet: Removed LNI 192.168.203.160@tcp107 [12512.736154] LNet: Removed LNI 192.168.203.160@tcp108 [12513.760194] LNet: Removed LNI 192.168.203.160@tcp109 [12514.784146] LNet: Removed LNI 192.168.203.160@tcp110 [12514.785464] LNetError: 467179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12514.787295] LNetError: 467179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12514.789619] LNet: Removed LNI 192.168.203.160@tcp111 [12515.808197] LNet: Removed LNI 192.168.203.160@tcp112 [12516.832145] LNet: Removed LNI 192.168.203.160@tcp113 [12517.856163] LNet: Removed LNI 192.168.203.160@tcp114 [12518.880166] LNet: Removed LNI 192.168.203.160@tcp115 [12519.904166] LNet: Removed LNI 192.168.203.160@tcp116 [12520.928170] LNet: Removed LNI 192.168.203.160@tcp117 [12521.952147] LNet: Removed LNI 192.168.203.160@tcp118 [12522.976157] LNet: Removed LNI 192.168.203.160@tcp119 [12522.977729] LNetError: 467179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12522.980514] LNetError: 467179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [12522.983531] LNet: Removed LNI 192.168.203.160@tcp120 [12524.000128] LNet: Removed LNI 192.168.203.160@tcp121 [12524.001590] LNet: Removed LNI 192.168.203.160@tcp122 [12525.024103] LNet: Removed LNI 192.168.203.160@tcp123 [12526.048139] LNet: Removed LNI 192.168.203.160@tcp124 [12527.072118] LNet: Removed LNI 192.168.203.160@tcp125 [12528.096127] LNet: Removed LNI 192.168.203.160@tcp126 [12529.120118] LNet: Removed LNI 192.168.203.160@tcp127 [12530.144114] LNet: Removed LNI 192.168.203.160@tcp128 [12531.168103] LNet: Removed LNI 192.168.203.160@tcp129 [12531.169694] LNet: Removed LNI 192.168.203.160@tcp130 [12531.171047] LNet: Removed LNI 192.168.203.160@tcp131 [12531.172502] LNet: Removed LNI 192.168.203.160@tcp132 [12531.174067] LNet: Removed LNI 192.168.203.160@tcp133 [12531.175255] LNet: Removed LNI 192.168.203.160@tcp134 [12531.182425] LNet: Removed LNI 192.168.203.160@tcp135 [12531.441974] Key type .llcrypt unregistered [12531.443468] Key type ._llcrypt unregistered [12538.484975] Key type ._llcrypt registered [12538.486389] Key type .llcrypt registered [12538.526434] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [12544.801351] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12545.182267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12545.189614] alg: No test for adler32 (adler32-zlib) [12546.088000] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [12546.193058] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [12547.784193] Key type lgssc registered [12548.221384] Lustre: Echo OBD driver; http://www.lustre.org/ [12559.295893] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 08:06:04 (1788696364) [12565.108506] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [12566.151787] Key type lgssc unregistered [12566.277299] LNet: 471234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12566.280280] LNetError: 471234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12566.290302] LNet: Removed LNI 192.168.203.160@tcp [12566.576285] Key type .llcrypt unregistered [12566.577913] Key type ._llcrypt unregistered [12573.027248] Key type ._llcrypt registered [12573.028375] Key type .llcrypt registered [12573.064821] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12573.536639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12573.553328] alg: No test for adler32 (adler32-zlib) [12574.396625] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [12574.477259] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [12576.056143] Key type lgssc registered [12576.407226] Lustre: Echo OBD driver; http://www.lustre.org/ [12579.277479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12579.282285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12580.374121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12580.384572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12580.423102] Lustre: lustre-MDT0000: new disk, initializing [12580.445156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12580.452287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12581.740204] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12585.792802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12585.837151] Lustre: 472421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12585.858873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12585.860694] Lustre: Skipped 1 previous similar message [12585.896898] Lustre: lustre-MDT0001: new disk, initializing [12585.928623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12585.939898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12585.943090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12587.287551] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12590.779837] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12592.533738] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12594.715840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12594.811289] Lustre: lustre-OST0000: new disk, initializing [12594.812800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12594.815105] Lustre: 474385:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12594.840213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12596.720778] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12600.314493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12600.318341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12600.355376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12600.507868] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12601.631957] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing lnet_if_list [12603.154275] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12603.166165] LNet: Added LNI 192.168.203.160@tcp10 [8/256/0/180] [12604.737180] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12604.747130] LNet: Added LNI 192.168.203.160@tcp11 [8/256/0/180] [12605.998188] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12606.013091] LNet: Added LNI 192.168.203.160@tcp12 [8/256/0/180] [12607.390247] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12607.400739] LNet: Added LNI 192.168.203.160@tcp13 [8/256/0/180] [12608.788537] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12608.800451] LNet: Added LNI 192.168.203.160@tcp14 [8/256/0/180] [12610.252567] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12610.262829] LNet: Added LNI 192.168.203.160@tcp15 [8/256/0/180] [12611.755161] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12611.768720] LNet: Added LNI 192.168.203.160@tcp16 [8/256/0/180] [12613.283451] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12613.294743] LNet: Added LNI 192.168.203.160@tcp17 [8/256/0/180] [12614.819877] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12614.832542] LNet: Added LNI 192.168.203.160@tcp18 [8/256/0/180] [12616.445862] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12616.456801] LNet: Added LNI 192.168.203.160@tcp19 [8/256/0/180] [12618.100324] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12618.111648] LNet: Added LNI 192.168.203.160@tcp20 [8/256/0/180] [12619.744598] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12619.756808] LNet: Added LNI 192.168.203.160@tcp21 [8/256/0/180] [12621.520619] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12621.532581] LNet: Added LNI 192.168.203.160@tcp22 [8/256/0/180] [12623.274269] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12623.285810] LNet: Added LNI 192.168.203.160@tcp23 [8/256/0/180] [12625.115873] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12625.128553] LNet: Added LNI 192.168.203.160@tcp24 [8/256/0/180] [12627.122883] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12627.134939] LNet: Added LNI 192.168.203.160@tcp25 [8/256/0/180] [12628.851879] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12628.865410] LNet: Added LNI 192.168.203.160@tcp26 [8/256/0/180] [12630.541149] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12630.552498] LNet: Added LNI 192.168.203.160@tcp27 [8/256/0/180] [12632.333818] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12632.345365] LNet: Added LNI 192.168.203.160@tcp28 [8/256/0/180] [12634.266157] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12634.277164] LNet: Added LNI 192.168.203.160@tcp29 [8/256/0/180] [12636.200710] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12636.214775] LNet: Added LNI 192.168.203.160@tcp30 [8/256/0/180] [12638.168917] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12638.180229] LNet: Added LNI 192.168.203.160@tcp31 [8/256/0/180] [12640.039376] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12640.049200] LNet: Added LNI 192.168.203.160@tcp32 [8/256/0/180] [12641.865233] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12641.877840] LNet: Added LNI 192.168.203.160@tcp33 [8/256/0/180] [12643.740547] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12643.752415] LNet: Added LNI 192.168.203.160@tcp34 [8/256/0/180] [12645.675986] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12645.685896] LNet: Added LNI 192.168.203.160@tcp35 [8/256/0/180] [12647.735588] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12647.750113] LNet: Added LNI 192.168.203.160@tcp36 [8/256/0/180] [12649.823017] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12649.833345] LNet: Added LNI 192.168.203.160@tcp37 [8/256/0/180] [12652.069574] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12652.079772] LNet: Added LNI 192.168.203.160@tcp38 [8/256/0/180] [12654.502053] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12654.513789] LNet: Added LNI 192.168.203.160@tcp39 [8/256/0/180] [12656.819141] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12656.831722] LNet: Added LNI 192.168.203.160@tcp40 [8/256/0/180] [12659.188298] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12659.200372] LNet: Added LNI 192.168.203.160@tcp41 [8/256/0/180] [12661.504231] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12661.517531] LNet: Added LNI 192.168.203.160@tcp42 [8/256/0/180] [12663.870737] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12663.881965] LNet: Added LNI 192.168.203.160@tcp43 [8/256/0/180] [12666.325260] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12666.337133] LNet: Added LNI 192.168.203.160@tcp44 [8/256/0/180] [12668.636119] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12668.647471] LNet: Added LNI 192.168.203.160@tcp45 [8/256/0/180] [12670.908731] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12670.919497] LNet: Added LNI 192.168.203.160@tcp46 [8/256/0/180] [12673.357472] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12673.367282] LNet: Added LNI 192.168.203.160@tcp47 [8/256/0/180] [12675.781589] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12675.792250] LNet: Added LNI 192.168.203.160@tcp48 [8/256/0/180] [12678.278057] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12678.290934] LNet: Added LNI 192.168.203.160@tcp49 [8/256/0/180] [12680.937688] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12680.949538] LNet: Added LNI 192.168.203.160@tcp50 [8/256/0/180] [12688.352941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12688.356699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12688.363203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12692.449419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12692.450514] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12692.456352] Lustre: Skipped 1 previous similar message [12694.229893] Lustre: server umount lustre-MDT0000 complete [12695.377313] LustreError: 473213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788696501 with bad export cookie 1365496210740827621 [12695.377810] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12695.381228] LustreError: 473213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12695.494595] Lustre: server umount lustre-MDT0001 complete [12702.363029] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12705.281473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12713.952136] Lustre: 471692:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788696504/real 1788696504] req@00000000176a21e9 x1875584096484224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788696520 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12713.960891] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12717.024101] Lustre: 471691:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788696507/real 1788696507] req@0000000069e68918 x1875584096484352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788696523 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12719.072109] Lustre: 471693:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788696509/real 1788696509] req@0000000026a89702 x1875584096484480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788696525 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12720.096300] LustreError: 471690:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bd05f313 x1875584096485376/t0(0) o250->MGC192.168.203.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12720.192548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12721.248101] Lustre: 471694:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788696511/real 1788696511] req@0000000089ab6aa2 x1875584096484736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788696527 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12721.409478] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12724.035130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12725.157518] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12725.339767] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12726.240125] Lustre: 471691:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788696516/real 1788696516] req@00000000587fa41c x1875584096484864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788696532 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12727.666211] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12729.552067] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12733.349064] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12738.530272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12738.531371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12738.536013] Lustre: Skipped 1 previous similar message [12738.539810] Lustre: Skipped 3 previous similar messages [12743.196591] Lustre: server umount lustre-OST0000 complete [12744.672758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12744.674894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12744.681354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12748.768914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12748.770935] Lustre: Skipped 1 previous similar message [12750.483092] Lustre: server umount lustre-MDT0000 complete [12751.690202] LustreError: 472413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788696558 with bad export cookie 1365496210740829658 [12751.693129] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12751.695260] LustreError: 472413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12751.795723] Lustre: server umount lustre-MDT0001 complete [12755.230445] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [12756.227484] Key type lgssc unregistered [12756.378654] LNet: 501339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12756.382378] LNetError: 501339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12756.385030] LNet: Removed LNI 192.168.203.160@tcp [12756.386402] LNet: Removed LNI 192.168.203.160@tcp10 [12756.387802] LNet: Removed LNI 192.168.203.160@tcp11 [12756.389227] LNet: Removed LNI 192.168.203.160@tcp12 [12756.390570] LNet: Removed LNI 192.168.203.160@tcp13 [12756.392087] LNet: Removed LNI 192.168.203.160@tcp14 [12756.393644] LNet: Removed LNI 192.168.203.160@tcp15 [12756.395073] LNet: Removed LNI 192.168.203.160@tcp16 [12756.396258] LNet: Removed LNI 192.168.203.160@tcp17 [12756.397664] LNet: Removed LNI 192.168.203.160@tcp18 [12756.398706] LNet: Removed LNI 192.168.203.160@tcp19 [12756.400080] LNet: Removed LNI 192.168.203.160@tcp20 [12756.401327] LNet: Removed LNI 192.168.203.160@tcp21 [12756.402398] LNet: Removed LNI 192.168.203.160@tcp22 [12756.403787] LNet: Removed LNI 192.168.203.160@tcp23 [12756.404986] LNet: Removed LNI 192.168.203.160@tcp24 [12756.406360] LNet: Removed LNI 192.168.203.160@tcp25 [12756.408366] LNet: Removed LNI 192.168.203.160@tcp26 [12756.409967] LNet: Removed LNI 192.168.203.160@tcp27 [12756.411586] LNet: Removed LNI 192.168.203.160@tcp28 [12756.413279] LNet: Removed LNI 192.168.203.160@tcp29 [12756.414346] LNet: Removed LNI 192.168.203.160@tcp30 [12756.415912] LNet: Removed LNI 192.168.203.160@tcp31 [12756.417582] LNet: Removed LNI 192.168.203.160@tcp32 [12756.419258] LNet: Removed LNI 192.168.203.160@tcp33 [12756.421082] LNet: Removed LNI 192.168.203.160@tcp34 [12756.422713] LNet: Removed LNI 192.168.203.160@tcp35 [12756.423976] LNet: Removed LNI 192.168.203.160@tcp36 [12756.425186] LNet: Removed LNI 192.168.203.160@tcp37 [12756.426921] LNet: Removed LNI 192.168.203.160@tcp38 [12756.428645] LNet: Removed LNI 192.168.203.160@tcp39 [12756.430311] LNet: Removed LNI 192.168.203.160@tcp40 [12756.431971] LNet: Removed LNI 192.168.203.160@tcp41 [12756.433718] LNet: Removed LNI 192.168.203.160@tcp42 [12756.435340] LNet: Removed LNI 192.168.203.160@tcp43 [12756.437065] LNet: Removed LNI 192.168.203.160@tcp44 [12756.438716] LNet: Removed LNI 192.168.203.160@tcp45 [12756.440242] LNet: Removed LNI 192.168.203.160@tcp46 [12756.441631] LNet: Removed LNI 192.168.203.160@tcp47 [12756.442848] LNet: Removed LNI 192.168.203.160@tcp48 [12756.443963] LNet: Removed LNI 192.168.203.160@tcp49 [12756.452447] LNet: Removed LNI 192.168.203.160@tcp50 [12756.714700] Key type .llcrypt unregistered [12756.716034] Key type ._llcrypt unregistered [12762.966832] Key type ._llcrypt registered [12762.967961] Key type .llcrypt registered [12763.003548] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [12769.083532] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12769.438478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12769.445725] alg: No test for adler32 (adler32-zlib) [12770.305492] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [12770.402707] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [12771.984155] Key type lgssc registered [12772.347131] Lustre: Echo OBD driver; http://www.lustre.org/ [12783.646772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12783.651314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12784.727521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12784.737161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12784.772762] Lustre: lustre-MDT0000: new disk, initializing [12784.797315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12784.804691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12786.229904] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12790.479088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12790.524811] Lustre: 504520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12790.549728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12790.552764] Lustre: Skipped 1 previous similar message [12790.589923] Lustre: lustre-MDT0001: new disk, initializing [12790.624136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12790.636309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12790.641957] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12791.976202] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12795.958643] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12797.902716] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12800.474267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12800.562733] Lustre: *** cfs_fail_loc=90f, val=0*** [12800.599080] Lustre: lustre-OST0000: new disk, initializing [12800.600658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12800.603664] Lustre: 506538:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12800.635563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12802.447350] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12806.149118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12806.153255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12806.156672] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12806.198496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12811.235617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12811.235778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12811.238053] LustreError: Skipped 1 previous similar message [12811.242020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12811.245072] Lustre: Skipped 1 previous similar message [12814.546637] Lustre: server umount lustre-OST0000 complete [12816.353488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12816.353525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12816.354484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12816.361096] Lustre: Skipped 2 previous similar messages [12821.472954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12821.474836] Lustre: Skipped 2 previous similar messages [12826.593501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12826.595700] Lustre: Skipped 1 previous similar message [12830.176108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12830.245407] Lustre: server umount lustre-MDT0000 complete [12831.532164] LustreError: 504512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788696638 with bad export cookie 173526925281452780 [12831.534919] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12831.536574] LustreError: 504512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12831.655326] Lustre: server umount lustre-MDT0001 complete [12835.574201] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [12836.679724] Key type lgssc unregistered [12836.813540] LNet: 508440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12836.817596] LNetError: 508440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12836.829625] LNet: Removed LNI 192.168.203.160@tcp [12837.161742] Key type .llcrypt unregistered [12837.163711] Key type ._llcrypt unregistered [12844.912848] Key type ._llcrypt registered [12844.914114] Key type .llcrypt registered [12844.950188] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12845.288551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12845.328230] alg: No test for adler32 (adler32-zlib) [12846.177698] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [12846.261817] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [12847.840196] Key type lgssc registered [12848.191638] Lustre: Echo OBD driver; http://www.lustre.org/ [12850.656898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12850.664071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12851.793461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12853.038721] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12855.780151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12855.904772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12857.279905] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12859.725159] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12861.493385] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12863.767273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12863.872527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12865.753038] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12868.156845] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12882.257824] loop0: detected capacity change from 0 to 204800000 [12882.331430] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12882.335769] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12882.340513] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12884.405904] loop0: detected capacity change from 0 to 204800000 [12884.465950] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12884.469795] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12884.473392] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12886.263572] loop0: detected capacity change from 0 to 204800000 [12888.115617] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12888.161775] Lustre: fs15246-OST0000: new disk, initializing [12888.163778] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12888.165888] Lustre: 513297:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12888.183476] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12890.119212] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12893.951049] loop1: detected capacity change from 0 to 204800000 [12895.695738] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12895.730055] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12895.749472] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12895.781584] Lustre: fs15246-MDT0000: new disk, initializing [12895.823722] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12895.832656] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12895.867934] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12896.928732] Lustre: 514418:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000016679dea x1875584380600576/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788696719 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12897.122347] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12906.016291] LustreError: 514418:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [12906.020572] LustreError: 514418:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [12906.024189] LustreError: 514418:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [12909.029286] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12909.034540] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12910.109367] Lustre: server umount fs15246-MDT0000 complete [12913.095560] Lustre: server umount fs15246-OST0000 complete [12917.216975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12917.220174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12917.224842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12919.266121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12919.266460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12919.267289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12919.274426] Lustre: Skipped 1 previous similar message [12922.517293] Lustre: server umount lustre-OST0000 complete [12924.385318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12924.386087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12924.390048] Lustre: Skipped 1 previous similar message [12929.505774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12929.508082] Lustre: Skipped 2 previous similar messages [12938.208135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12938.277458] Lustre: server umount lustre-MDT0000 complete [12939.646570] LustreError: 514412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788696746 with bad export cookie 13837869289788831545 [12939.648618] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12939.653628] LustreError: 514412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12939.776108] Lustre: server umount lustre-MDT0001 complete [12943.320848] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [12944.373730] Key type lgssc unregistered [12944.502331] LNet: 517511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12944.505608] LNetError: 517511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12944.518299] LNet: Removed LNI 192.168.203.160@tcp [12944.836740] Key type .llcrypt unregistered [12944.837779] Key type ._llcrypt unregistered [12952.069688] Key type ._llcrypt registered [12952.070752] Key type .llcrypt registered [12952.107973] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12952.411611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12952.441057] alg: No test for adler32 (adler32-zlib) [12953.281653] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [12953.360688] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [12954.936087] Key type lgssc registered [12955.283521] Lustre: Echo OBD driver; http://www.lustre.org/ [12966.905960] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [12969.930053] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12980.944380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12980.950118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12982.021947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12982.031130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12982.064666] Lustre: lustre-MDT0000: new disk, initializing [12982.082520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12982.088714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12983.293857] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12987.418943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12987.460444] Lustre: 521956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12987.488288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12987.490326] Lustre: Skipped 1 previous similar message [12987.527162] Lustre: lustre-MDT0001: new disk, initializing [12987.562675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12987.578472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12987.582796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12988.946172] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12992.584554] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12994.327298] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12996.474545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12996.572016] Lustre: lustre-OST0000: new disk, initializing [12996.573520] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12996.575987] Lustre: 523923:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12996.602972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12998.486964] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [13000.252000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13000.255194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13000.298399] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13002.181122] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13003.548446] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13003.613989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13004.954964] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13005.039483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13005.792780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13005.796716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13005.801757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13010.402321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13010.403041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13010.407079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13011.790329] Lustre: server umount lustre-OST0000 complete [13013.472908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13013.476101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13013.480153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13013.481827] Lustre: Skipped 1 previous similar message [13015.521712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13015.521802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13020.640974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13020.642828] Lustre: Skipped 2 previous similar messages [13027.296104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13027.359790] Lustre: server umount lustre-MDT0000 complete [13028.522221] LustreError: 521950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788696835 with bad export cookie 7282393966637037160 [13028.524664] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13028.527458] LustreError: 521950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13028.635286] Lustre: server umount lustre-MDT0001 complete [13036.111191] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [13039.325955] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [13049.714394] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12787 sec ========= 08:14:15 (1788696855) [13050.246950] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:14:15 (1788696855) === [13051.334550] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:14:16 (1788696856) === [13064.078873] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [13065.034576] Key type lgssc unregistered [13065.153293] LNet: 530551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13065.156104] LNetError: 530551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13065.162297] LNet: Removed LNI 192.168.203.160@tcp [13065.462216] Key type .llcrypt unregistered [13065.463740] Key type ._llcrypt unregistered