[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 548457052 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003372] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008427] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009011] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.011023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.012028] pid_max: default: 32768 minimum: 301 [ 0.014055] LSM: Security Framework initializing [ 0.015063] Yama: becoming mindful. [ 0.016042] SELinux: Initializing. [ 0.017093] *** VALIDATE selinux *** [ 0.027439] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.032607] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.034027] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036170] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.037161] *** VALIDATE tmpfs *** [ 0.039375] *** VALIDATE proc *** [ 0.041082] *** VALIDATE cgroup *** [ 0.042013] *** VALIDATE cgroup2 *** [ 0.044160] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.045208] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.047010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.048035] Spectre V2 : User space: Vulnerable [ 0.049016] Speculative Store Bypass: Vulnerable [ 0.052522] debug: unmapping init [mem 0xffffffff8c459000-0xffffffff8c460fff] [ 0.054282] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.055998] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.057035] ... version: 2 [ 0.058015] ... bit width: 48 [ 0.059013] ... generic registers: 4 [ 0.060013] ... value mask: 0000ffffffffffff [ 0.061012] ... max period: 00007fffffffffff [ 0.062018] ... fixed-purpose events: 3 [ 0.063012] ... event mask: 000000070000000f [ 0.064381] rcu: Hierarchical SRCU implementation. [ 0.066786] smp: Bringing up secondary CPUs ... [ 0.067686] x86: Booting SMP configuration: [ 0.068029] .... node #0, CPUs: #1 #2 #3 [ 0.072014] smp: Brought up 1 node, 4 CPUs [ 0.074017] smpboot: Max logical packages: 1 [ 0.075012] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.121641] node 0 deferred pages initialised in 44ms [ 0.126237] devtmpfs: initialized [ 0.128337] x86/mm: Memory block size: 128MB [ 0.132000] gcov: version magic: 0x41383552 [ 0.133121] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.135172] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.138531] pinctrl core: initialized pinctrl subsystem [ 0.141224] [ 0.141868] ************************************************************* [ 0.145013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148012] ** ** [ 0.150131] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153013] ** ** [ 0.156020] ** This means that this kernel is built to expose internal ** [ 0.158011] ** IOMMU data structures, which may compromise security on ** [ 0.161015] ** your system. ** [ 0.163013] ** ** [ 0.166019] ** If you see this message and you are not debugging the ** [ 0.169147] ** kernel, report this immediately to your vendor! ** [ 0.172021] ** ** [ 0.175013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178015] ************************************************************* [ 0.181857] NET: Registered protocol family 16 [ 0.184478] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.187063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.191074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196585] cpuidle: using governor menu [ 0.199996] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202716] PCI: Using configuration type 1 for base access [ 0.206161] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.215161] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.217017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220786] cryptd: max_cpu_qlen set to 1000 [ 0.222215] ACPI: Added _OSI(Module Device) [ 0.224102] ACPI: Added _OSI(Processor Device) [ 0.225015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.227016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.231667] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.239189] ACPI: Interpreter enabled [ 0.240106] ACPI: PM: (supports S0 S3 S4 S5) [ 0.241015] ACPI: Using IOAPIC for interrupt routing [ 0.242235] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.246529] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.259182] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261093] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263086] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.267116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273060] acpiphp: Slot [2] registered [ 0.274238] acpiphp: Slot [5] registered [ 0.276175] acpiphp: Slot [6] registered [ 0.278221] acpiphp: Slot [7] registered [ 0.280256] acpiphp: Slot [8] registered [ 0.282155] acpiphp: Slot [9] registered [ 0.283164] acpiphp: Slot [10] registered [ 0.285158] acpiphp: Slot [3] registered [ 0.286211] acpiphp: Slot [4] registered [ 0.288180] acpiphp: Slot [11] registered [ 0.289253] acpiphp: Slot [12] registered [ 0.292288] acpiphp: Slot [13] registered [ 0.293140] acpiphp: Slot [14] registered [ 0.295098] acpiphp: Slot [15] registered [ 0.296111] acpiphp: Slot [16] registered [ 0.297101] acpiphp: Slot [17] registered [ 0.299144] acpiphp: Slot [18] registered [ 0.302221] acpiphp: Slot [19] registered [ 0.304304] acpiphp: Slot [20] registered [ 0.306205] acpiphp: Slot [21] registered [ 0.310372] acpiphp: Slot [22] registered [ 0.313196] acpiphp: Slot [23] registered [ 0.314198] acpiphp: Slot [24] registered [ 0.316176] acpiphp: Slot [25] registered [ 0.318261] acpiphp: Slot [26] registered [ 0.320305] acpiphp: Slot [27] registered [ 0.321206] acpiphp: Slot [28] registered [ 0.323179] acpiphp: Slot [29] registered [ 0.325265] acpiphp: Slot [30] registered [ 0.327268] acpiphp: Slot [31] registered [ 0.329135] PCI host bridge to bus 0000:00 [ 0.331024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.333056] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.336040] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.346044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.349272] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.352249] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.357000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.367920] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.374549] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.377036] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.381051] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.386029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.390628] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.393998] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.396038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.397900] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.401018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.419021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.426017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.433827] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.441018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.448019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.500019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.512000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.522020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.529014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.555026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.579032] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.592016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.604013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.632015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.646000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.655021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.665021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.695020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.717609] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.727012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.746021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.767017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.779899] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.788024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.797067] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.816033] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.829074] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.831373] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.833413] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.837412] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.839230] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.844093] iommu: Default domain type: Passthrough [ 0.845484] SCSI subsystem initialized [ 0.847106] ACPI: bus type USB registered [ 0.848145] usbcore: registered new interface driver usbfs [ 0.850075] usbcore: registered new interface driver hub [ 0.852110] usbcore: registered new device driver usb [ 0.854169] pps_core: LinuxPPS API ver. 1 registered [ 0.855009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.859101] PTP clock support registered [ 0.861086] EDAC MC: Ver: 3.0.0 [ 0.863062] PCI: Using ACPI for IRQ routing [ 0.864572] NetLabel: Initializing [ 0.866012] NetLabel: domain hash size = 128 [ 0.867010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.869093] NetLabel: unlabeled traffic allowed by default [ 0.871338] vgaarb: loaded [ 0.873323] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.875013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.882000] clocksource: Switched to clocksource kvm-clock [ 0.998701] VFS: Disk quotas dquot_6.6.0 [ 1.000322] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.002797] *** VALIDATE ramfs *** [ 1.004165] *** VALIDATE hugetlbfs *** [ 1.005673] pnp: PnP ACPI init [ 1.008200] pnp: PnP ACPI: found 6 devices [ 1.026546] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.030038] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.033027] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.035074] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.037757] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.040095] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.043016] NET: Registered protocol family 2 [ 1.045304] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.049820] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.053270] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.058429] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.061799] TCP: Hash tables configured (established 65536 bind 65536) [ 1.064454] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.067369] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.070322] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.073576] NET: Registered protocol family 1 [ 1.075996] RPC: Registered named UNIX socket transport module. [ 1.078357] RPC: Registered udp transport module. [ 1.080034] RPC: Registered tcp transport module. [ 1.081677] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.084188] NET: Registered protocol family 44 [ 1.085844] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.088126] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.090143] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.092574] PCI: CLS 0 bytes, default 64 [ 1.095124] Unpacking initramfs... [ 2.650941] debug: unmapping init [mem 0xffff9830bcc54000-0xffff9830bffbffff] [ 2.654191] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.655751] software IO TLB: mapped [mem 0x00000000a2e00000-0x00000000a6e00000] (64MB) [ 2.657808] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 3.165186] Initialise system trusted keyrings [ 3.166655] Key type blacklist registered [ 3.168644] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.178524] zbud: loaded [ 3.181347] *** VALIDATE nfs *** [ 3.182594] *** VALIDATE nfs4 *** [ 3.184381] pstore: using deflate compression [ 3.188214] Platform Keyring initialized [ 3.299792] NET: Registered protocol family 38 [ 3.301779] Key type asymmetric registered [ 3.303210] Asymmetric key parser 'x509' registered [ 3.305138] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.308295] io scheduler mq-deadline registered [ 3.310024] io scheduler kyber registered [ 3.311560] io scheduler bfq registered [ 3.313640] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.316328] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.318869] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.321852] ACPI: Power Button [PWRF] [ 3.327713] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.334822] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.355916] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.371079] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.390756] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.419650] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.449615] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.454878] Non-volatile memory driver v1.3 [ 3.457069] Linux agpgart interface v0.103 [ 3.493568] virtio_blk virtio1: [vda] 145168 512-byte logical blocks (74.3 MB/70.9 MiB) [ 3.497014] vda: detected capacity change from 0 to 74326016 [ 3.510953] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.513858] vdb: detected capacity change from 0 to 1073741824 [ 3.528755] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.533264] vdc: detected capacity change from 0 to 2621440000 [ 3.547910] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.550990] vdd: detected capacity change from 0 to 2621440000 [ 3.567532] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.570802] vde: detected capacity change from 0 to 4294967296 [ 3.588420] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.592148] vdf: detected capacity change from 0 to 4294967296 [ 3.599553] libphy: Fixed MDIO Bus: probed [ 3.604481] usbcore: registered new interface driver usbserial_generic [ 3.607595] usbserial: USB Serial support registered for generic [ 3.610496] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.615287] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.617233] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.619594] mousedev: PS/2 mouse device common for all mice [ 3.623296] rtc_cmos 00:05: RTC can wake from S4 [ 3.628581] rtc_cmos 00:05: registered as rtc0 [ 3.630420] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.632145] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.636787] intel_pstate: CPU model not supported [ 3.644587] hid: raw HID events driver (C) Jiri Kosina [ 3.649617] usbcore: registered new interface driver usbhid [ 3.651589] usbhid: USB HID core driver [ 3.654314] drop_monitor: Initializing network drop monitor service [ 3.656238] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.659649] Initializing XFRM netlink socket [ 3.660346] NET: Registered protocol family 10 [ 3.671656] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.677526] Segment Routing with IPv6 [ 3.679132] NET: Registered protocol family 17 [ 3.681937] mpls_gso: MPLS GSO support [ 3.689905] RAS: Correctable Errors collector initialized. [ 3.692444] AVX version of gcm_enc/dec engaged. [ 3.694366] AES CTR mode by8 optimization enabled [ 3.774880] sched_clock: Marking stable (3774857861, 0)->(4753158940, -978301079) [ 3.781718] registered taskstats version 1 [ 3.783722] Loading compiled-in X.509 certificates [ 3.786162] zswap: loaded using pool lzo/zbud [ 3.810986] Key type big_key registered [ 3.824490] Key type encrypted registered [ 3.826348] ima: No TPM chip found, activating TPM-bypass! [ 3.828832] ima: Allocated hash algorithm: sha1 [ 3.830844] ima: No architecture policies found [ 3.833125] evm: Initialising EVM extended attributes: [ 3.835307] evm: security.selinux [ 3.836926] evm: security.ima [ 3.838228] evm: security.capability [ 3.839843] evm: HMAC attrs: 0x1 [ 3.842592] rtc_cmos 00:05: setting system clock to 2026-07-04 03:18:31 UTC (1783135111) [ 3.849185] debug: unmapping init [mem 0xffffffff8d403000-0xffffffff8d5fffff] [ 3.852944] debug: unmapping init [mem 0xffffffff8c182000-0xffffffff8c458fff] [ 3.862078] Write protecting the kernel read-only data: 28672k [ 3.866182] debug: unmapping init [mem 0xffffffff8a803000-0xffffffff8a9fffff] [ 3.870331] debug: unmapping init [mem 0xffffffff8b114000-0xffffffff8b1fffff] [ 3.906358] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.911684] systemd[1]: Detected virtualization kvm. [ 3.913370] systemd[1]: Detected architecture x86-64. [ 3.915886] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.945949] systemd[1]: No hostname configured. [ 3.947650] systemd[1]: Set hostname to . [ 3.950068] random: systemd: uninitialized urandom read (16 bytes read) [ 3.953096] systemd[1]: Initializing machine ID from random generator. [ 4.088497] random: systemd: uninitialized urandom read (16 bytes read) [ 4.091072] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.095547] random: systemd: uninitialized urandom read (16 bytes read) [ 4.098297] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.103409] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.927614] device-mapper: uevent: version 1.0.3 [ 4.930232] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 5.807649] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.956420] virtio_net virtio0 ens2: renamed from eth0 [ 5.981687] scsi host0: ata_piix [ 5.993726] scsi host1: ata_piix [ 5.996369] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.998634] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.349225] random: crng init done [ 10.352114] random: 7 urandom warning(s) missed due to ratelimiting [ 10.373688] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.192048] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.758191] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.075935] SELinux: Disabled at runtime. [ 13.136813] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.150306] systemd[1]: Detected virtualization kvm. [ 13.153633] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.857829] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.862439] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.869630] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.875041] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.879034] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.894232] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.906154] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 14.043720] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 14.500461] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 14.957699] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.031891] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.684343] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.730811] EDAC sbridge: Ver: 1.1.2 [ 18.359090] Key type dns_resolver registered [ 18.677139] NFS: Registering the id_resolver key type [ 18.679186] Key type id_resolver registered [ 18.680838] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started OpenSSH server daemon. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg245-server login: [ 45.763122] libcfs: loading out-of-tree module taints kernel. [ 45.786937] Key type ._llcrypt registered [ 45.788752] Key type .llcrypt registered [ 45.847056] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 60.492860] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 62.490505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.541620] alg: No test for adler32 (adler32-zlib) [ 64.271310] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 65.257303] hrtimer: interrupt took 23289569 ns [ 65.648943] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 67.519367] Key type lgssc registered [ 69.862701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 89.873848] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 132.300549] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 144.492586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 144.510842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 145.771903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 145.805292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 145.922730] Lustre: lustre-MDT0000: new disk, initializing [ 146.022242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 146.042585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 150.758865] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.971608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.065958] Lustre: 6503:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 164.091909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 164.098354] Lustre: Skipped 1 previous similar message [ 164.174903] Lustre: lustre-MDT0001: new disk, initializing [ 164.251824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.277828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 164.287888] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 168.961682] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.497460] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 182.659500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 183.026297] Lustre: lustre-OST0000: new disk, initializing [ 183.030649] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 183.039904] Lustre: 8441:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 183.140745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.544960] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.981085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 190.003815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 190.151086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.362555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.465150] Lustre: lustre-OST0001: new disk, initializing [ 204.468929] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 204.474058] Lustre: 9514:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.543189] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.526518] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.632359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 213.640561] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 213.700675] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 223.436925] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 230.556332] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 237.979122] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing check_logdir /tmp/testlogs/ [ 243.292459] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing yml_node [ 248.941827] Lustre: DEBUG MARKER: Client: 2.17.54.84 [ 251.639402] Lustre: DEBUG MARKER: MDS: 2.17.54.84 [ 254.859034] Lustre: DEBUG MARKER: OSS: 2.17.54.84 [ 256.359222] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jul 3 23:22:41 EDT 2026 [ 277.869973] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 279.801721] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 295.904962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.907441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 295.915057] Lustre: Skipped 1 previous similar message [ 295.929669] Lustre: Skipped 3 previous similar messages [ 301.023892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.043716] Lustre: Skipped 3 previous similar messages [ 301.934456] Lustre: server umount lustre-MDT0000 complete [ 305.568992] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783135413 with bad export cookie 9507598261497593601 [ 305.578350] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 305.579727] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 306.145133] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.162893] Lustre: Skipped 2 previous similar messages [ 306.167651] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 306.174652] Lustre: Skipped 1 previous similar message [ 310.178424] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.181391] Lustre: Skipped 1 previous similar message [ 312.206717] Lustre: server umount lustre-MDT0001 complete [ 316.784510] Lustre: server umount lustre-OST0000 complete [ 320.641677] Lustre: server umount lustre-OST0001 complete [ 327.575309] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 338.583720] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 377.970436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 378.266661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 378.289276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 378.414905] Lustre: lustre-MDT0000: new disk, initializing [ 378.519977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 378.559772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 383.801530] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 396.823694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 397.003464] Lustre: 15064:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 397.073247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 397.078298] Lustre: Skipped 1 previous similar message [ 397.240126] Lustre: lustre-MDT0001: new disk, initializing [ 397.415900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 397.445280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 397.461465] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 402.971885] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 413.178935] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 420.310564] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 427.848952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 428.247600] Lustre: lustre-OST0000: new disk, initializing [ 428.252392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 428.264339] Lustre: 17035:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 428.379128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.718125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 429.730773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 429.967962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 435.509943] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 446.811779] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 452.990603] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 453.425694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 459.234391] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 459.605990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 465.896574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.900898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 465.913097] Lustre: Skipped 2 previous similar messages [ 465.914831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 465.925149] LustreError: Skipped 1 previous similar message [ 465.944154] Lustre: Skipped 1 previous similar message [ 469.157790] Lustre: server umount lustre-OST0000 complete [ 476.141043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.145584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 476.163108] Lustre: Skipped 1 previous similar message [ 481.257736] Lustre: server umount lustre-MDT0000 complete [ 481.259153] LustreError: 15069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.298844] LustreError: 15069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 485.352424] LustreError: 15070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.381439] LustreError: 15070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 486.152370] LustreError: 15862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783135593 with bad export cookie 9507598261497595134 [ 486.156035] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 486.179066] LustreError: 15862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 486.881621] Lustre: server umount lustre-MDT0001 complete [ 492.477606] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 23:26:37 (1783135597) [ 495.396738] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 497.681849] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 23:26:42 (1783135602) [ 504.522725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 505.340225] LustreError: 19446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.454736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.950852] LustreError: 19447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.963534] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 516.070070] LustreError: 19446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.262440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.732769] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 534.490416] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 541.010533] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 551.268853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 559.717439] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 570.616985] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 589.790770] Lustre: server umount lustre-OST0000 complete [ 595.941940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.950615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 595.965394] Lustre: Skipped 1 previous similar message [ 595.990020] Lustre: Skipped 2 previous similar messages [ 605.272182] Lustre: server umount lustre-MDT0000 complete [ 606.182964] LustreError: 19441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.224719] LustreError: 19441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 607.200364] Lustre: 3635:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783135698/real 1783135698] req@ffff98311814b100 x1869752747343872/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783135714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 607.250252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 608.287228] Lustre: 3637:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783135699/real 1783135699] req@ffff98311738a300 x1869752747344256/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1783135715 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 610.167069] LustreError: 19427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783135717 with bad export cookie 9507598261497596016 [ 610.180220] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 610.181328] LustreError: 19427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 610.674081] Lustre: server umount lustre-MDT0001 complete [ 624.268337] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 627.670882] Key type lgssc unregistered [ 628.007526] LNet: 23000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.024829] LNetError: 23000:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 628.049232] LNet: Removed LNI 192.168.202.145@tcp [ 629.382291] Key type .llcrypt unregistered [ 629.387234] Key type ._llcrypt unregistered [ 655.910410] Key type ._llcrypt registered [ 655.918233] Key type .llcrypt registered [ 656.100359] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 657.771890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 657.823041] alg: No test for adler32 (adler32-zlib) [ 659.085098] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 659.626306] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 661.399185] Key type lgssc registered [ 662.923944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 676.600308] loop: module loaded [ 676.728947] loop0: detected capacity change from 0 to 204800000 [ 677.129718] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.166818] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.750843] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.284222] loop0: detected capacity change from 0 to 204800000 [ 685.435662] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.485302] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.503352] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 692.352899] loop0: detected capacity change from 0 to 204800000 [ 699.351822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 699.394038] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 700.845029] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 700.948124] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 701.047299] Lustre: lustre-MDT0000: new disk, initializing [ 701.152839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.179684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 706.025753] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 715.106737] loop1: detected capacity change from 0 to 204800000 [ 721.012193] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 721.299894] Lustre: lustre-OST07c6: new disk, initializing [ 721.308721] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 721.317318] Lustre: Skipped 1 previous similar message [ 721.331828] Lustre: 26191:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 721.444145] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 727.637145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 727.643111] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 727.677991] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 728.708545] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 733.684449] Lustre: 25177:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 733.697625] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 733.703347] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 733.707080] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 733.714814] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 733.720154] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 750.486948] Lustre: 25177:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 750.498325] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 750.504097] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 750.511619] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 750.516626] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 750.521602] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 755.416082] Lustre: 26706:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 755.434632] Lustre: 26706:0:(osd_internal.h:1444:osd_trans_exec_op()) Skipped 1 previous similar message [ 755.444708] Lustre: 26706:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 755.453625] Lustre: 26706:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.464936] Lustre: 26706:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 755.475512] Lustre: 26706:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.486016] Lustre: 26706:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 755.495050] Lustre: 26706:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.509116] Lustre: 26706:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 755.522608] Lustre: 26706:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.533380] Lustre: 26706:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 755.548161] Lustre: 26706:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.791094] Lustre: 25177:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 758.797435] Lustre: 25177:0:(osd_internal.h:1444:osd_trans_exec_op()) Skipped 1 previous similar message [ 758.801820] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 758.805536] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.809249] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 758.812604] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.815438] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 758.818665] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.821891] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 758.826849] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.832236] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 758.837407] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.040832] Lustre: 25179:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 764.048054] Lustre: 25179:0:(osd_internal.h:1444:osd_trans_exec_op()) Skipped 1 previous similar message [ 764.054195] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 764.060680] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.070150] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 764.078385] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.082892] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 764.096431] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.103409] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 764.111726] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.120395] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 764.128301] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.088990] Lustre: 25179:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 775.099746] Lustre: 25179:0:(osd_internal.h:1444:osd_trans_exec_op()) Skipped 5 previous similar messages [ 775.105083] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 775.110776] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 775.116589] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 775.122688] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 775.128741] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 775.135757] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 775.148812] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 775.155580] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 775.159363] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 775.165642] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 824.800898] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 824.807272] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 824.825001] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 827.876338] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 832.991927] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 837.087153] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 837.343067] Lustre: server umount lustre-OST07c6 complete [ 847.338951] Lustre: server umount lustre-MDT0000 complete [ 863.947281] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 866.748687] Key type lgssc unregistered [ 867.090337] LNet: 28752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.096896] LNetError: 28752:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.110130] LNet: Removed LNI 192.168.202.145@tcp [ 867.995428] Key type .llcrypt unregistered [ 867.997440] Key type ._llcrypt unregistered [ 887.847283] Key type ._llcrypt registered [ 887.852611] Key type .llcrypt registered [ 888.011329] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 889.244400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 889.381979] alg: No test for adler32 (adler32-zlib) [ 890.391685] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 890.637689] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 892.295387] Key type lgssc registered [ 893.736799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 900.843851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 900.857205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 902.255045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.494888] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 911.843979] LustreError: 30024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 911.872865] LustreError: 30024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 916.929756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.964972] LustreError: 30025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 917.374833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 921.988706] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 929.781916] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 936.666424] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 943.378823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 943.762050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.860315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 950.401100] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 959.119237] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 983.010502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.033673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 983.045311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 988.134470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 988.153966] Lustre: Skipped 1 previous similar message [ 988.567387] Lustre: server umount lustre-OST0000 complete [ 993.250822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.264922] Lustre: Skipped 1 previous similar message [ 993.285330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 993.296507] Lustre: Skipped 2 previous similar messages [ 998.369503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 998.379062] Lustre: Skipped 1 previous similar message [ 999.558497] Lustre: server umount lustre-MDT0000 complete [ 1003.157081] LustreError: 30007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783136110 with bad export cookie 2023678933138393297 [ 1003.166488] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1003.170291] LustreError: 30007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1003.628926] Lustre: server umount lustre-MDT0001 complete [ 1018.013119] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1021.900541] Key type lgssc unregistered [ 1022.292911] LNet: 33760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.299876] LNetError: 33760:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.325405] LNet: Removed LNI 192.168.202.145@tcp [ 1023.574195] Key type .llcrypt unregistered [ 1023.576476] Key type ._llcrypt unregistered [ 1043.486627] Key type ._llcrypt registered [ 1043.491120] Key type .llcrypt registered [ 1043.602152] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1044.510471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1044.610222] alg: No test for adler32 (adler32-zlib) [ 1045.659388] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 1045.871468] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1047.551302] Key type lgssc registered [ 1048.760551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1057.071593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1057.112174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1058.670792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1063.138812] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1069.028292] LustreError: 35029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1069.057510] LustreError: 35029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1071.507100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1071.937048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1076.773922] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1084.701385] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1091.596700] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1099.387374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1099.714859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1103.876284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1106.248839] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1113.693945] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1129.440441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1129.448312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.472732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1132.512764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1132.516175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1132.523484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.542541] Lustre: Skipped 1 previous similar message [ 1134.194064] Lustre: server umount lustre-OST0000 complete [ 1138.663831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1138.674992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.692330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.752839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.755509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.782461] Lustre: Skipped 1 previous similar message [ 1144.571793] Lustre: server umount lustre-MDT0000 complete [ 1147.874300] LustreError: 35024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1147.892750] LustreError: 35024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1148.194305] LustreError: 35010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783136255 with bad export cookie 7258040630865461152 [ 1148.219333] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1148.221476] LustreError: 35010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1148.666923] Lustre: server umount lustre-MDT0001 complete [ 1160.825704] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1165.011783] Key type lgssc unregistered [ 1165.679717] LNet: 38587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.697636] LNetError: 38587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1166.761206] LNet: Removed LNI 192.168.202.145@tcp [ 1168.334641] Key type .llcrypt unregistered [ 1168.337976] Key type ._llcrypt unregistered [ 1194.904949] Key type ._llcrypt registered [ 1194.907558] Key type .llcrypt registered [ 1194.987445] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1195.848943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1195.877563] alg: No test for adler32 (adler32-zlib) [ 1196.970431] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 1197.167724] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1198.847137] Key type lgssc registered [ 1200.007950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1207.238205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1207.254324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.538688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1212.133651] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1219.042758] LustreError: 40004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.062683] LustreError: 40004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1219.715568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1220.064621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1224.576449] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1232.013900] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1236.914165] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1243.413563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1243.807349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.729610] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1254.035501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1261.994661] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1269.619535] Lustre: server umount lustre-MDT0000 complete [ 1269.729709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.733777] LustreError: 40005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.740414] Lustre: Skipped 2 previous similar messages [ 1269.770598] LustreError: 40005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1274.128752] LustreError: 39985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783136381 with bad export cookie 15391562690004562056 [ 1274.131349] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1274.140300] LustreError: 39985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1274.848457] LustreError: 41017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.850631] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1274.870372] LustreError: 41017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1274.881915] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1276.703300] Lustre: server umount lustre-MDT0001 complete [ 1306.079652] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1306.284187] Lustre: server umount lustre-OST0000 complete [ 1326.547914] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1330.530548] Key type lgssc unregistered [ 1330.781790] LNet: 43865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1330.788529] LNetError: 43865:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.800065] LNet: Removed LNI 192.168.202.145@tcp [ 1331.974634] Key type .llcrypt unregistered [ 1331.977740] Key type ._llcrypt unregistered [ 1356.511558] Key type ._llcrypt registered [ 1356.514444] Key type .llcrypt registered [ 1356.619713] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1357.589929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1357.622114] alg: No test for adler32 (adler32-zlib) [ 1358.671301] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 1358.949520] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1360.639153] Key type lgssc registered [ 1361.946714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1370.844877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1370.860196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.246562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1377.091748] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1381.347904] LustreError: 45283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.361577] LustreError: 45283:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1385.557233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1385.859373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1390.506464] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1398.638541] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1405.285833] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1412.903593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1413.317060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1416.367652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1420.264942] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1428.826647] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1441.033625] Lustre: lustre-OST0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 1441.250795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1441.261433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1444.119720] Lustre: server umount lustre-OST0000 complete [ 1466.857490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.895548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1466.921496] Lustre: Skipped 2 previous similar messages [ 1466.946852] Lustre: Skipped 3 previous similar messages [ 1469.566595] Lustre: server umount lustre-MDT0000 complete [ 1471.989840] LustreError: 47775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1472.017944] LustreError: 47775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1474.446610] LustreError: 45262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783136582 with bad export cookie 18249449147586087448 [ 1474.451715] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1474.480870] LustreError: 45262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1481.140519] Lustre: server umount lustre-MDT0001 complete [ 1495.560421] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1499.690777] Key type lgssc unregistered [ 1500.035034] LNet: 48990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.048188] LNetError: 48990:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.089640] LNet: Removed LNI 192.168.202.145@tcp [ 1501.351922] Key type .llcrypt unregistered [ 1501.354545] Key type ._llcrypt unregistered [ 1529.102261] Key type ._llcrypt registered [ 1529.104229] Key type .llcrypt registered [ 1529.237528] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1531.022828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1531.124843] alg: No test for adler32 (adler32-zlib) [ 1532.189821] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 1532.477084] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1534.199255] Key type lgssc registered [ 1535.864700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1547.003565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1547.035684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.098251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1555.987204] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1558.505590] LustreError: 50409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.540828] LustreError: 50409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1563.619961] LustreError: 50410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1567.077405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1567.548810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1572.411771] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1581.317308] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1587.997995] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1596.142363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1596.482189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1603.137566] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1611.248875] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1620.035134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1626.947615] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1630.887836] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1643.505318] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-0323h45m48s [ 1645.295884] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1647.861106] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 1648.935933] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 1649.632616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1649.640885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1650.723136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1650.738923] Lustre: Skipped 6 previous similar messages [ 1653.996093] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 1654.249948] Lustre: server umount lustre-MDT0000 complete [ 1656.827071] LustreError: 51140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.860801] LustreError: 51140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1657.541087] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1660.139342] LustreError: 50406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1664.183330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1664.364916] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1664.632107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.737759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1665.676263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1670.420679] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1675.115436] Lustre: DEBUG MARKER: Wait for df (36553) ... [ 1677.192974] Lustre: DEBUG MARKER: done [ 1684.448801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1684.459534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1684.472370] Lustre: Skipped 2 previous similar messages [ 1684.482246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1687.818575] Lustre: server umount lustre-OST0000 complete [ 1691.618921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1691.636211] Lustre: Skipped 1 previous similar message [ 1691.638193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1691.646230] LustreError: Skipped 1 previous similar message [ 1692.706246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1692.723624] Lustre: Skipped 6 previous similar messages [ 1697.689948] Lustre: server umount lustre-MDT0000 complete [ 1697.771645] LustreError: 50404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1697.790226] LustreError: 50404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1701.652352] LustreError: 50392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783136809 with bad export cookie 8609395200763159300 [ 1701.660387] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1701.668870] LustreError: 50392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1701.981859] Lustre: server umount lustre-MDT0001 complete [ 1713.148467] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1716.222581] Key type lgssc unregistered [ 1716.474704] LNet: 55357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.478762] LNetError: 55357:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1716.495746] LNet: Removed LNI 192.168.202.145@tcp [ 1717.328077] Key type .llcrypt unregistered [ 1717.330645] Key type ._llcrypt unregistered [ 1762.903560] Key type ._llcrypt registered [ 1762.906740] Key type .llcrypt registered [ 1763.047382] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1764.540141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1764.567512] alg: No test for adler32 (adler32-zlib) [ 1765.671748] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 1765.918016] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1767.607172] Key type lgssc registered [ 1768.465848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1776.088176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1776.140161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1777.468398] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1777.496037] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1777.741472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1782.553990] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1791.201637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.274267] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1791.295849] Lustre: 57568:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1791.507773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1796.848851] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1804.700467] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1810.815418] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1817.710609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1818.035409] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1818.251679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.639125] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1826.444117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1833.449847] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1840.443578] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-0323h49m06s [ 1842.265515] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1849.207453] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1851.547144] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1851.604623] LustreError: 57576:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1872.157579] Lustre: lustre-MDT0000: Client 0a08e98c-8dc0-4011-b29c-459292ddd94b (at 192.168.202.45@tcp) reconnecting [ 1891.655179] LustreError: 57576:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1894.801511] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1897.719884] Lustre: DEBUG MARKER: done [ 1908.192432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1908.199738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.222421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1908.705282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1908.726225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.757534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1911.495490] Lustre: server umount lustre-OST0000 complete [ 1918.946249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.947305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1918.960477] Lustre: Skipped 1 previous similar message [ 1918.984354] Lustre: Skipped 1 previous similar message [ 1921.641034] Lustre: server umount lustre-MDT0000 complete [ 1926.041579] LustreError: 57560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137033 with bad export cookie 11682156501221690759 [ 1926.042872] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1926.053290] LustreError: 57560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1926.484608] Lustre: server umount lustre-MDT0001 complete [ 1940.025271] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1942.795399] Key type lgssc unregistered [ 1943.191869] LNet: 61902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.202899] LNetError: 61902:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.222616] LNet: Removed LNI 192.168.202.145@tcp [ 1944.109985] Key type .llcrypt unregistered [ 1944.112853] Key type ._llcrypt unregistered [ 1997.132483] Key type ._llcrypt registered [ 1997.135480] Key type .llcrypt registered [ 1997.248547] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1998.899506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1998.931767] alg: No test for adler32 (adler32-zlib) [ 2000.007565] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 2000.398176] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2002.096131] Key type lgssc registered [ 2003.370837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2006.957775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2006.992157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.422721] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2008.485415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2008.661849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.662475] Lustre: Failing over lustre-MDT0000 [ 2009.897442] Lustre: server umount lustre-MDT0000 complete [ 2018.053771] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:52:03 (1783137123) [ 2025.074878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.450307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2029.984438] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2039.507543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.615509] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2039.674244] Lustre: 64702:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2039.988737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2045.231168] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2053.396975] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2059.978424] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2068.846250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2068.983118] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2069.119522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.122908] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2077.332544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2084.173342] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2090.013296] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2098.144574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.148507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.169452] Lustre: Skipped 1 previous similar message [ 2098.180305] Lustre: Skipped 1 previous similar message [ 2103.230172] Lustre: server umount lustre-MDT0000 complete [ 2103.269311] LustreError: 67175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.310336] LustreError: 67175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2107.121505] LustreError: 64694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137214 with bad export cookie 8764500264327845413 [ 2107.131507] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2107.134355] LustreError: 64694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2107.352679] Lustre: server umount lustre-MDT0001 complete [ 2111.215113] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2116.407622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2124.774273] Lustre: 63624:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783137216/real 1783137216] req@000000001270b862 x1869754776199808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783137232 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2124.833175] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.862163] Lustre: Skipped 1 previous similar message [ 2127.673882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.929173] Lustre: 63625:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783137220/real 1783137220] req@0000000009dd889e x1869754776199936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783137236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2129.951242] Lustre: 63626:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783137221/real 1783137221] req@00000000f2a7ae9b x1869754776200064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783137237 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2132.063825] LustreError: 63623:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d1ba5aa4 x1869754776200832/t0(0) o250->MGC192.168.202.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2132.661211] LustreError: 66551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.796590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.951156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2133.792851] Lustre: 63627:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783137224/real 1783137224] req@00000000e0ba1e93 x1869754776200320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783137240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2133.795969] LustreError: 66552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.952552] Lustre: 63625:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783137229/real 1783137229] req@000000006068d25f x1869754776200448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783137245 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2138.805187] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2140.131200] LustreError: 68471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2140.159223] LustreError: 68471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2145.261688] LustreError: 66551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2145.305904] LustreError: 66551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2148.626339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.162225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2153.598149] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2154.279591] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2161.552877] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2169.810805] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2180.066645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2180.070531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2180.077190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2180.111505] Lustre: Skipped 2 previous similar messages [ 2184.859861] Lustre: server umount lustre-MDT0000 complete [ 2185.193968] LustreError: 68469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2185.234129] LustreError: 68469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2188.853433] LustreError: 64694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137296 with bad export cookie 8764500264327847695 [ 2188.855552] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2188.860722] LustreError: 64694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2189.275750] Lustre: server umount lustre-MDT0001 complete [ 2195.988617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2205.599184] Lustre: 63627:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783137297/real 1783137297] req@00000000ce7f8698 x1869754776228480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783137313 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2205.659777] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2207.103467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.290689] LustreError: 66552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.443584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.570068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2220.284737] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2230.479343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.919851] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2234.934699] Lustre: Skipped 1 previous similar message [ 2235.627600] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2242.958579] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2248.531852] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2255.332981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.337361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2255.345559] Lustre: Skipped 2 previous similar messages [ 2255.353609] Lustre: Skipped 4 previous similar messages [ 2256.566476] Lustre: server umount lustre-MDT0000 complete [ 2260.449061] LustreError: 66551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.488056] LustreError: 66551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2261.213491] LustreError: 64694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137368 with bad export cookie 8764500264327849074 [ 2261.215579] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2261.219283] LustreError: 64694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2261.623708] Lustre: server umount lustre-MDT0001 complete [ 2270.326258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2274.622530] Lustre: DEBUG MARKER: files compared the same [ 2288.356849] Lustre: server umount lustre-OST0000 complete [ 2306.771593] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2309.550078] Key type lgssc unregistered [ 2309.808733] LNet: 74431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.816556] LNetError: 74431:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2310.890389] LNet: Removed LNI 192.168.202.145@tcp [ 2311.704952] Key type .llcrypt unregistered [ 2311.707195] Key type ._llcrypt unregistered [ 2336.698440] Key type ._llcrypt registered [ 2336.701295] Key type .llcrypt registered [ 2336.924140] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2338.904780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2339.150033] alg: No test for adler32 (adler32-zlib) [ 2340.290655] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 2340.583634] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2342.375224] Key type lgssc registered [ 2343.813056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2353.257979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2353.289164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.969947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2359.818144] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug +malloc all [ 2364.388887] LustreError: 75851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.396532] LustreError: 75851:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2368.825042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.113266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2373.562970] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug +malloc all [ 2381.308743] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2387.409622] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2395.034325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2395.557737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2399.689506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2402.661343] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug +malloc all [ 2413.812184] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2424.295052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2424.307453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.310359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2424.327260] Lustre: Skipped 1 previous similar message [ 2424.347888] Lustre: Skipped 1 previous similar message [ 2429.411333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2429.433300] Lustre: Skipped 1 previous similar message [ 2429.788389] Lustre: server umount lustre-OST0000 complete [ 2434.529985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.548222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2434.551948] Lustre: Skipped 1 previous similar message [ 2439.649887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2439.656853] Lustre: Skipped 1 previous similar message [ 2440.731267] Lustre: server umount lustre-MDT0000 complete [ 2444.770566] LustreError: 77597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.796574] LustreError: 77597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2445.327575] LustreError: 75831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137552 with bad export cookie 12641689028206794668 [ 2445.337813] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2445.345731] LustreError: 75831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2445.812047] Lustre: server umount lustre-MDT0001 complete [ 2458.640733] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2461.331423] Key type lgssc unregistered [ 2461.578735] LNet: 79388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2461.583925] LNetError: 79388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2461.604712] LNet: Removed LNI 192.168.202.145@tcp [ 2462.395938] Key type .llcrypt unregistered [ 2462.398276] Key type ._llcrypt unregistered [ 2487.405892] Key type ._llcrypt registered [ 2487.408918] Key type .llcrypt registered [ 2487.544576] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2488.600584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2488.803034] alg: No test for adler32 (adler32-zlib) [ 2489.977915] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 2490.168523] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2491.823450] Key type lgssc registered [ 2493.826824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2505.856236] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2505.886481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2521.631543] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2526.753988] LustreError: 80784:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 2553.503385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.793791] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2570.401589] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2578.650038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.849978] Lustre: *** cfs_fail_loc=706, val=0*** [ 2579.453551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.513374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2584.357972] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2584.569079] LustreError: 80809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.600911] LustreError: 80809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2587.694298] LustreError: 81881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2587.735710] LustreError: 81881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2592.747585] LustreError: 80810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.772601] LustreError: 80810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2593.510204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.829340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2598.658681] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2608.044694] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2614.693247] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2619.374694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.397714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2619.411241] Lustre: Skipped 1 previous similar message [ 2623.691637] Lustre: server umount lustre-OST0000 complete [ 2629.606890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.620286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2629.631752] Lustre: Skipped 1 previous similar message [ 2629.634345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2629.654117] Lustre: Skipped 1 previous similar message [ 2634.678402] Lustre: server umount lustre-MDT0000 complete [ 2634.726938] LustreError: 81883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.746028] LustreError: 81883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2638.925170] LustreError: 80791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137746 with bad export cookie 9297310890603458199 [ 2638.947286] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2639.198380] Lustre: server umount lustre-MDT0001 complete [ 2651.091490] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2654.411194] Key type lgssc unregistered [ 2654.870372] LNet: 84414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2654.876541] LNetError: 84414:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2654.914148] LNet: Removed LNI 192.168.202.145@tcp [ 2655.933982] Key type .llcrypt unregistered [ 2655.937209] Key type ._llcrypt unregistered [ 2689.621674] Key type ._llcrypt registered [ 2689.623945] Key type .llcrypt registered [ 2689.852975] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2692.637631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2692.667750] alg: No test for adler32 (adler32-zlib) [ 2694.080741] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 2694.351949] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2696.039442] Key type lgssc registered [ 2697.190298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2705.948993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2705.977864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.199250] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2722.124320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.622368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2727.426586] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2727.905684] LustreError: 86522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.927376] LustreError: 86522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2733.025124] LustreError: 86521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2733.052151] LustreError: 86521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2736.666404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2736.932772] LustreError: 86522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2736.955892] LustreError: 86522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2736.984883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.254160] LustreError: 86526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.271848] LustreError: 86526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2745.403268] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2747.366777] LustreError: 86522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2747.401222] LustreError: 86522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2754.611968] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2754.621059] Lustre: MGS: Client aecdd7f1-eb57-4a8e-b465-71185a266f7b (at 0@lo) reconnecting [ 2754.648855] LustreError: 86536:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2754.924602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2755.053193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2760.469606] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2775.527080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.539481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2775.544906] Lustre: Skipped 1 previous similar message [ 2775.562190] Lustre: Skipped 1 previous similar message [ 2779.973713] Lustre: server umount lustre-OST0000 complete [ 2785.761479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2785.767048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2785.772211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2785.780124] Lustre: Skipped 1 previous similar message [ 2790.402718] Lustre: server umount lustre-MDT0000 complete [ 2790.884432] LustreError: 87269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2790.913728] LustreError: 87269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2794.469089] Lustre: server umount lustre-MDT0001 complete [ 2798.320580] Lustre: server umount MGS complete [ 2810.265954] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 00:05:15 (1783137915) [ 2846.432785] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 2857.383321] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2895.507862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2901.219794] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2912.423460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2912.632334] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2912.664726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2912.744395] Lustre: lustre-MDT0001: new disk, initializing [ 2912.856900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2912.905235] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2912.908484] Lustre: 93692:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2912.935297] LustreError: 93692:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2918.108989] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2930.373060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2930.800228] Lustre: lustre-OST0000: new disk, initializing [ 2930.805526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2930.811348] Lustre: 94458:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2930.921538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.880468] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2937.953536] Lustre: 94753:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000a1864bb9 x1869755503911296/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783138061 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2948.668645] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2955.447973] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2955.475472] Lustre: MGS: Client 11bb2073-4c4a-4483-b054-a9b581726b94 (at 0@lo) reconnecting [ 2955.552933] Lustre: 95504:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2955.565979] LustreError: 93678:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2955.604665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2955.708797] Lustre: lustre-MDT0000: new disk, initializing [ 2955.873041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2955.902393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2955.935745] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2956.962777] Lustre: 95529:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000462f6c86 x1869755503918208/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1783138080 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2961.480930] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2962.033869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2962.050693] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2962.244318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2991.080582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2991.092289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.142314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2991.148088] Lustre: Skipped 1 previous similar message [ 2997.735253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2997.737978] Lustre: Skipped 2 previous similar messages [ 3002.849542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3002.856256] Lustre: Skipped 1 previous similar message [ 3003.373296] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3003.544989] Lustre: server umount lustre-OST0000 complete [ 3013.094915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3013.096057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3013.110325] Lustre: Skipped 1 previous similar message [ 3013.112723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3013.124430] Lustre: Skipped 1 previous similar message [ 3024.352437] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3024.480782] Lustre: server umount lustre-MDT0000 complete [ 3034.726231] Lustre: server umount lustre-MDT0001 complete [ 3039.189465] Lustre: server umount MGS complete [ 3050.951832] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:09:16 (1783138156) [ 3072.142259] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3075.357354] Key type lgssc unregistered [ 3075.945995] LNet: 98031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3075.951863] LNetError: 98031:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3075.994792] LNet: Removed LNI 192.168.202.145@tcp [ 3077.017212] Key type .llcrypt unregistered [ 3077.021930] Key type ._llcrypt unregistered [ 3098.908903] Key type ._llcrypt registered [ 3098.916413] Key type .llcrypt registered [ 3099.014794] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3100.836055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3100.874890] alg: No test for adler32 (adler32-zlib) [ 3101.962652] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 3102.313403] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3104.087267] Key type lgssc registered [ 3105.411965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3113.655355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3113.670119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.249395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3120.085729] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3124.729932] LustreError: 99258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.749330] LustreError: 99258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3128.978606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3129.275610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3134.138614] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3141.515510] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3148.340854] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3155.583954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3155.948932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3160.059352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3162.505834] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3170.051732] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3183.623610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3183.637165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3183.664627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3183.674580] Lustre: Skipped 1 previous similar message [ 3187.327593] Lustre: server umount lustre-OST0000 complete [ 3193.834675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3193.843571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3193.863452] Lustre: Skipped 2 previous similar messages [ 3193.885507] Lustre: Skipped 1 previous similar message [ 3198.944409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3198.955708] Lustre: Skipped 1 previous similar message [ 3199.059664] Lustre: server umount lustre-MDT0000 complete [ 3203.744579] LustreError: 99240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138311 with bad export cookie 5598470878555662034 [ 3203.746237] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3203.763304] LustreError: 99240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3204.289647] Lustre: server umount lustre-MDT0001 complete [ 3218.602260] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3220.740945] Key type lgssc unregistered [ 3221.029797] LNet: 102792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3221.040642] LNetError: 102792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3221.062863] LNet: Removed LNI 192.168.202.145@tcp [ 3221.789794] Key type .llcrypt unregistered [ 3221.793386] Key type ._llcrypt unregistered [ 3243.991102] Key type ._llcrypt registered [ 3243.996257] Key type .llcrypt registered [ 3244.116292] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3245.848125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3245.902300] alg: No test for adler32 (adler32-zlib) [ 3247.048886] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 3247.318294] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3249.103187] Key type lgssc registered [ 3250.676673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3258.030189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3258.034758] LustreError: 103877:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3258.038082] Lustre: Skipped 1 previous similar message [ 3258.038103] LustreError: 103876:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3258.058903] LustreError: 103877:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3258.063456] LustreError: 103877:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3258.070472] LustreError: 103877:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3258.079080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.735430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3268.596532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3268.990791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3274.212852] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3281.266725] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing lsmod [ 3285.297958] LustreError: 105085:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3285.297958] LustreError: 105084:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3285.318051] LustreError: 105084:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4979 [ 3285.322899] LustreError: 105084:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3285.338880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3285.350737] LustreError: 105084:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3285.781601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.578552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3289.596783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.615582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3291.105367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3291.111633] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3291.142869] Lustre: Skipped 1 previous similar message [ 3294.242587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3294.250346] Lustre: Skipped 4 previous similar messages [ 3295.874566] Lustre: server umount lustre-MDT0000 complete [ 3299.305588] LustreError: 103933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3299.325397] LustreError: 103933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3300.066389] LustreError: 103892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138407 with bad export cookie 13157538642303265933 [ 3300.073745] LustreError: 103892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3300.081972] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3300.407569] LustreError: 104347:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f152755e x1869756082705536/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3300.872544] Lustre: server umount lustre-MDT0001 complete [ 3314.793442] Lustre: server umount lustre-OST0000 complete [ 3325.040445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3325.513760] LustreError: 106149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3325.571669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.060931] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3331.048933] LustreError: 106150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3336.173633] LustreError: 106149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3340.746583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.260426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3346.717759] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3355.026520] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3361.574339] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3370.626620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3370.982630] LustreError: 107970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3371.091502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3376.124595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3379.818773] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3388.066482] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3396.586192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.586383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3396.588516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3396.609835] Lustre: Skipped 1 previous similar message [ 3396.652239] LustreError: Skipped 1 previous similar message [ 3400.805255] Lustre: server umount lustre-OST0000 complete [ 3406.828261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.837888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3406.841791] Lustre: Skipped 1 previous similar message [ 3406.877663] Lustre: Skipped 2 previous similar messages [ 3411.634966] Lustre: server umount lustre-MDT0000 complete [ 3411.938813] LustreError: 107101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3411.955389] LustreError: 107101:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3416.182208] LustreError: 106130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138523 with bad export cookie 13157538642303266633 [ 3416.189951] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3416.195934] LustreError: 106130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3416.459879] Lustre: server umount lustre-MDT0001 complete [ 3430.803107] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3434.109956] Key type lgssc unregistered [ 3434.514102] LNet: 109684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3434.525528] LNetError: 109684:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3434.549271] LNet: Removed LNI 192.168.202.145@tcp [ 3435.689758] Key type .llcrypt unregistered [ 3435.692411] Key type ._llcrypt unregistered [ 3463.378359] Key type ._llcrypt registered [ 3463.380958] Key type .llcrypt registered [ 3463.454718] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3464.564299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3464.648386] alg: No test for adler32 (adler32-zlib) [ 3465.719546] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 3466.017807] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3467.647400] Key type lgssc registered [ 3468.886371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3477.606698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3477.644215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3479.174692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.630670] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3488.838575] LustreError: 111105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3488.861706] LustreError: 111105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3493.876338] LustreError: 111106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3496.419095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.022318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3501.725969] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3510.220716] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3517.907719] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3525.891602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3526.248082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3533.210289] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3542.216554] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3548.790170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3557.876927] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3562.510733] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3567.600773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3567.608124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3567.623178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3569.636709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3569.640626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3569.677678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3572.957475] Lustre: server umount lustre-OST0000 complete [ 3578.848654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3578.871492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3578.882183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3578.884463] Lustre: Skipped 1 previous similar message [ 3583.360747] Lustre: server umount lustre-MDT0000 complete [ 3584.993164] LustreError: 111102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3585.019768] LustreError: 111102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3587.176481] LustreError: 111086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138694 with bad export cookie 11569204086889230945 [ 3587.190288] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3587.527733] Lustre: server umount lustre-MDT0001 complete [ 3599.628199] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3602.572288] Key type lgssc unregistered [ 3602.930299] LNet: 114766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3602.937130] LNetError: 114766:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3602.966727] LNet: Removed LNI 192.168.202.145@tcp [ 3604.006195] Key type .llcrypt unregistered [ 3604.009078] Key type ._llcrypt unregistered [ 3623.620717] Key type ._llcrypt registered [ 3623.630751] Key type .llcrypt registered [ 3623.795028] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3624.931575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3624.951782] alg: No test for adler32 (adler32-zlib) [ 3626.340291] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 3626.623388] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3628.343185] Key type lgssc registered [ 3629.587428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3637.615780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3637.640591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3639.431780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3646.050450] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3649.077403] LustreError: 115988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3649.109421] LustreError: 115988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3654.124618] LustreError: 115988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3657.583948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3657.896370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3663.062134] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3671.818327] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3677.605379] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3685.482643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3685.911461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3692.438371] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3700.623888] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3709.507193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3719.650181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3719.656924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3719.667611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3719.673405] Lustre: Skipped 1 previous similar message [ 3724.776090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3724.786092] Lustre: Skipped 2 previous similar messages [ 3725.513195] Lustre: server umount lustre-OST0000 complete [ 3729.899895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3729.903344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3729.904573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3729.915773] Lustre: Skipped 1 previous similar message [ 3735.015218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3735.028308] Lustre: Skipped 2 previous similar messages [ 3735.541520] Lustre: server umount lustre-MDT0000 complete [ 3739.361182] LustreError: 115969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138847 with bad export cookie 15748531016199431593 [ 3739.374209] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3739.733677] Lustre: server umount lustre-MDT0001 complete [ 3749.967911] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3752.196776] Key type lgssc unregistered [ 3752.408914] LNet: 119544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3752.415367] LNetError: 119544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3753.445788] LNet: Removed LNI 192.168.202.145@tcp [ 3754.284316] Key type .llcrypt unregistered [ 3754.290647] Key type ._llcrypt unregistered [ 3777.112600] Key type ._llcrypt registered [ 3777.115201] Key type .llcrypt registered [ 3777.206790] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3778.335891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3778.398705] alg: No test for adler32 (adler32-zlib) [ 3779.370952] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 3779.632439] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3781.271587] Key type lgssc registered [ 3782.467732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3791.002446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3791.020063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3792.583923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3799.127837] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3803.106159] LustreError: 120958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3803.118340] LustreError: 120958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3808.238396] LustreError: 120957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3808.907686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.348534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3813.429550] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3820.008771] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3825.727862] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3832.178797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3832.465864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3837.686481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3838.374549] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3847.050047] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3853.388977] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3856.616706] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3856.619795] Lustre: Skipped 2 previous similar messages [ 3860.793184] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3860.799838] Lustre: Skipped 2 previous similar messages [ 3861.554994] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3861.584067] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3867.995616] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.45@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3868.003356] Lustre: Skipped 2 previous similar messages [ 3868.069181] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.45@tcp 192.168.0.[2,10]@tcp [ 3873.939976] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.45/24@tcp in log lustre-MDT0000 [ 3873.944748] Lustre: Skipped 2 previous similar messages [ 3875.677369] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.45/24@tcp [ 3875.687483] Lustre: Skipped 1 previous similar message [ 3884.004261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3884.005601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3884.012088] Lustre: Skipped 1 previous similar message [ 3884.016717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3889.127913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3889.134063] Lustre: Skipped 2 previous similar messages [ 3894.242767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3894.250532] Lustre: Skipped 1 previous similar message [ 3898.336604] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3898.515907] Lustre: server umount lustre-OST0000 complete [ 3904.489892] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3904.491593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3904.500916] Lustre: Skipped 1 previous similar message [ 3909.607161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3909.616868] Lustre: Skipped 2 previous similar messages [ 3916.255230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3916.433482] Lustre: server umount lustre-MDT0000 complete [ 3919.842169] LustreError: 120957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3919.862346] LustreError: 120957:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3920.126744] LustreError: 120939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139027 with bad export cookie 16780436962670912173 [ 3920.128870] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3920.139353] LustreError: 120939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3920.454931] Lustre: server umount lustre-MDT0001 complete [ 3932.212320] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3935.205272] Key type lgssc unregistered [ 3935.553829] LNet: 125556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3935.567983] LNetError: 125556:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3935.586929] LNet: Removed LNI 192.168.202.145@tcp [ 3936.416178] Key type .llcrypt unregistered [ 3936.419217] Key type ._llcrypt unregistered [ 3968.759888] Key type ._llcrypt registered [ 3968.761264] Key type .llcrypt registered [ 3968.827453] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3969.889954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3969.911955] alg: No test for adler32 (adler32-zlib) [ 3970.905713] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 3971.097141] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3972.759234] Key type lgssc registered [ 3973.858105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3980.769573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3980.797902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3982.125077] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3982.142364] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.45/24@tcp [ 3982.177026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3986.564576] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3991.527500] LustreError: 127118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.550266] LustreError: 127118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3993.510684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3993.778674] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3993.809740] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.45/24@tcp [ 3993.913826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3997.483766] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4004.929689] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4010.959371] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4017.011779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4017.285346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4022.926576] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4026.661535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4030.271991] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4042.208668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4042.218983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.257168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4042.269082] Lustre: Skipped 1 previous similar message [ 4047.217930] Lustre: server umount lustre-OST0000 complete [ 4052.449477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4052.453626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4052.458679] Lustre: Skipped 2 previous similar messages [ 4057.575602] LustreError: 127117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4057.595863] LustreError: 127117:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4057.790694] Lustre: server umount lustre-MDT0000 complete [ 4061.614423] LustreError: 127100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139169 with bad export cookie 16468104700319872499 [ 4061.622753] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4061.646957] LustreError: 127100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4062.063500] Lustre: server umount lustre-MDT0001 complete [ 4074.440667] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4077.051286] Key type lgssc unregistered [ 4077.316879] LNet: 130705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4077.331545] LNetError: 130705:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4077.362255] LNet: Removed LNI 192.168.202.145@tcp [ 4078.576508] Key type .llcrypt unregistered [ 4078.588456] Key type ._llcrypt unregistered [ 4108.109856] Key type ._llcrypt registered [ 4108.111702] Key type .llcrypt registered [ 4108.270731] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 4126.075348] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4127.084089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4127.356669] alg: No test for adler32 (adler32-zlib) [ 4128.683152] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 4128.934563] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4130.679731] Key type lgssc registered [ 4132.236511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4173.281812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4173.321135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.685687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4174.722569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4174.790166] Lustre: lustre-MDT0000: new disk, initializing [ 4174.855379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4174.873537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4179.976151] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4191.261083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4191.434179] Lustre: 134108:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4191.494305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4191.498534] Lustre: Skipped 1 previous similar message [ 4191.628405] Lustre: lustre-MDT0001: new disk, initializing [ 4191.743666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4191.777907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4191.791667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4196.680995] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4206.205177] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4212.869945] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4219.809537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4220.090347] Lustre: lustre-OST0000: new disk, initializing [ 4220.093156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4220.098687] Lustre: 136074:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4220.196750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4227.163503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4227.181499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4227.395254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4227.633667] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4237.776850] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4243.185702] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4243.444517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4248.287390] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4248.520114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4253.160529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4253.171920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4253.182804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4253.182294] LustreError: Skipped 1 previous similar message [ 4253.233772] Lustre: Skipped 1 previous similar message [ 4256.920425] Lustre: server umount lustre-OST0000 complete [ 4263.413472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4263.415973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.427844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4263.427856] Lustre: Skipped 1 previous similar message [ 4263.471504] Lustre: Skipped 1 previous similar message [ 4267.851344] Lustre: server umount lustre-MDT0000 complete [ 4271.571794] LustreError: 134099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139379 with bad export cookie 16998909736333092758 [ 4271.594313] LustreError: 134099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4271.607046] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4271.924472] Lustre: server umount lustre-MDT0001 complete [ 4280.690538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4281.284469] LustreError: 138292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4281.308930] LustreError: 138292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4281.397520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4286.147828] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4286.435394] LustreError: 138293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4290.538425] LustreError: 138293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4295.657854] LustreError: 138292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4296.355919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4296.796597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4301.546812] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4309.514540] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4315.251427] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4321.160164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4321.561870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4328.432358] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4335.071164] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4339.252269] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4339.456757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4344.920608] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4345.115869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4354.004220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4354.178980] Lustre: lustre-OST0001: new disk, initializing [ 4354.182453] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4354.187963] Lustre: 141665:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4354.285911] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4360.594723] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4361.346892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4361.358607] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4361.461340] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4369.735573] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4369.953862] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4375.388030] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4375.691630] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4381.702768] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff913b4c291800.ost_server_uuid 50 [ 4383.663438] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff913b4c291800.ost_server_uuid in IDLE state after 0 sec [ 4395.414679] Lustre: server umount lustre-OST0001 complete [ 4397.548720] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.560278] LustreError: 140116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4397.568908] Lustre: Skipped 1 previous similar message [ 4397.606813] LustreError: 140116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4402.660855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4402.671962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4402.672179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4402.672183] Lustre: Skipped 1 previous similar message [ 4402.680034] Lustre: Skipped 1 previous similar message [ 4406.625757] Lustre: server umount lustre-OST0000 complete [ 4412.897861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4412.901055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4412.927223] Lustre: Skipped 1 previous similar message [ 4412.931229] Lustre: Skipped 1 previous similar message [ 4416.954717] Lustre: server umount lustre-MDT0000 complete [ 4418.018456] LustreError: 138287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4418.047555] LustreError: 138287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4420.759566] LustreError: 138272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139528 with bad export cookie 16998909736333093640 [ 4420.760587] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4420.768733] LustreError: 138272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4421.218940] Lustre: server umount lustre-MDT0001 complete [ 4443.815275] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4446.727095] Key type lgssc unregistered [ 4446.980549] LNet: 144666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4446.985615] LNetError: 144666:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4447.004421] LNet: Removed LNI 192.168.202.145@tcp [ 4447.823481] Key type .llcrypt unregistered [ 4447.828379] Key type ._llcrypt unregistered [ 4491.433560] Key type ._llcrypt registered [ 4491.435824] Key type .llcrypt registered [ 4491.524715] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 4506.799632] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4508.743723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4508.782775] alg: No test for adler32 (adler32-zlib) [ 4509.975776] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 4510.310363] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4512.103254] Key type lgssc registered [ 4513.206949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4551.270342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4551.324454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4552.626568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4552.650699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4552.718570] Lustre: lustre-MDT0000: new disk, initializing [ 4552.943882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4552.992182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4558.737648] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4568.772367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4568.988222] Lustre: 148822:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4569.142472] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4569.154943] Lustre: Skipped 1 previous similar message [ 4569.389875] Lustre: lustre-MDT0001: new disk, initializing [ 4569.534248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4569.593134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4569.611488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4574.932465] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4585.170837] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4591.943717] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4599.493079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4599.803225] Lustre: lustre-OST0000: new disk, initializing [ 4599.810982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4599.827133] Lustre: 150792:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4599.950875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4606.029229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4606.048214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4606.212266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4607.932153] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4620.304674] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4627.251994] Lustre: Failing over lustre-OST0000 [ 4627.460231] Lustre: server umount lustre-OST0000 complete [ 4644.324513] Lustre: 147046:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139735/real 1783139735] req@00000000563542d2 x1869757407071616/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1783139751 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4644.354943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4646.896673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4647.179361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4647.212822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.230165] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4647.378210] Lustre: 147048:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139739/real 1783139739] req@000000008c96eda1 x1869757407072768/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783139755 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4649.252503] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4653.343083] Lustre: 147048:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139744/real 1783139744] req@00000000aba68451 x1869757407073920/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783139760 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4653.369027] Lustre: 147048:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4653.375696] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4654.395727] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4654.396647] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4657.096086] Lustre: Failing over lustre-MDT0000 [ 4657.313581] Lustre: server umount lustre-MDT0000 complete [ 4658.144668] Lustre: 147047:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139749/real 1783139749] req@00000000496ddde7 x1869757407075072/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783139765 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4658.189181] Lustre: 147047:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4659.682702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4659.702600] Lustre: Skipped 2 previous similar messages [ 4664.549881] LustreError: 148828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.561433] LustreError: 148828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4669.672563] LustreError: 148830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4669.693390] LustreError: 148830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4674.803936] LustreError: 151086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4674.832267] LustreError: 151086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4675.573925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4675.696569] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4675.902952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4675.959406] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4679.908208] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4680.058691] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4681.202682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4681.207297] Lustre: Skipped 1 previous similar message [ 4681.222160] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4681.275044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4690.407027] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4690.420659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4690.447517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4691.430972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4691.440712] Lustre: Skipped 1 previous similar message [ 4694.638290] Lustre: server umount lustre-OST0000 complete [ 4701.685882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4701.687690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4701.701766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4701.726607] Lustre: Skipped 2 previous similar messages [ 4701.746362] Lustre: Skipped 1 previous similar message [ 4704.498065] Lustre: server umount lustre-MDT0000 complete [ 4706.785582] LustreError: 148829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4706.804701] LustreError: 148829:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4708.357421] LustreError: 148815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139816 with bad export cookie 17127151493902696133 [ 4708.363591] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4708.381062] LustreError: 148815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4708.790616] Lustre: server umount lustre-MDT0001 complete [ 4720.155669] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4723.035840] Key type lgssc unregistered [ 4723.379202] LNet: 154445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4723.389308] LNetError: 154445:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4723.409256] LNet: Removed LNI 192.168.202.145@tcp [ 4724.414639] Key type .llcrypt unregistered [ 4724.417186] Key type ._llcrypt unregistered [ 4767.669735] Key type ._llcrypt registered [ 4767.672831] Key type .llcrypt registered [ 4767.828714] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4769.189653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4769.228053] alg: No test for adler32 (adler32-zlib) [ 4770.534931] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 4770.972740] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4772.655344] Key type lgssc registered [ 4773.682687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4781.410657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4781.435331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4782.826486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4788.033886] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4793.317519] LustreError: 155967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4793.360849] LustreError: 155967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4798.318254] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4798.432489] LustreError: 155968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4803.565484] LustreError: 155967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4803.900675] LustreError: 155996:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4803.929425] LustreError: 155996:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4803.977535] LustreError: 155996:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4804.958666] Lustre: server umount lustre-MDT0000 complete [ 4823.214935] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 4834.254973] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4874.028207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4874.378899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4874.509627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4874.646657] Lustre: lustre-MDT0000: new disk, initializing [ 4874.777395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4874.813086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4880.288914] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4892.569722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4892.757972] Lustre: 159301:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4892.850699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4892.854475] Lustre: Skipped 1 previous similar message [ 4892.938454] Lustre: lustre-MDT0001: new disk, initializing [ 4893.075821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4893.152588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4893.169770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4899.482573] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4909.670584] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4916.644226] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4923.777928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4924.170460] Lustre: lustre-OST0000: new disk, initializing [ 4924.176665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4924.192659] Lustre: 161271:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4924.283624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4930.544552] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4933.214352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4933.229246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4933.310542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4940.176882] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4944.648747] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4946.575924] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 00:40:51 (1783140051) [ 5210.593172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5210.608980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5210.619322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5213.174163] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5213.178141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5213.191108] Lustre: Skipped 1 previous similar message [ 5213.212473] Lustre: Skipped 2 previous similar messages [ 5214.620150] Lustre: server umount lustre-MDT0000 complete [ 5218.273650] LustreError: 159311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5218.286460] LustreError: 159311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5218.742768] LustreError: 159291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140326 with bad export cookie 8046241673424357314 [ 5218.743680] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5218.753059] LustreError: 159291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5223.392102] LustreError: 160434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5223.393278] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.426022] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5228.511702] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5233.119224] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5233.419452] Lustre: server umount lustre-MDT0001 complete [ 5243.873504] Lustre: server umount lustre-OST0000 complete [ 5254.892560] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 5266.699790] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5305.192935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5305.456165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5305.477375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5305.538771] Lustre: lustre-MDT0000: new disk, initializing [ 5305.591965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5305.608574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5310.458445] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5322.046224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5322.158461] Lustre: 165947:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5322.214982] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5322.219272] Lustre: Skipped 1 previous similar message [ 5322.339475] Lustre: lustre-MDT0001: new disk, initializing [ 5322.474937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5322.530664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5322.540720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5326.988911] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5336.041638] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5341.563846] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5347.418512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5347.646780] Lustre: lustre-OST0000: new disk, initializing [ 5347.650733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5347.655762] Lustre: 167914:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5347.713798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5353.483831] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5356.077393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5356.094189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5356.176337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5363.279567] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5368.058161] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5368.282799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5372.488119] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5372.668551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5377.007231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5377.011326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5377.011466] LustreError: Skipped 1 previous similar message [ 5377.018737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5377.018745] Lustre: Skipped 1 previous similar message [ 5377.054859] Lustre: Skipped 1 previous similar message [ 5380.946063] Lustre: server umount lustre-OST0000 complete [ 5387.240602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5387.262265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5387.263130] Lustre: Skipped 1 previous similar message [ 5387.289772] Lustre: Skipped 2 previous similar messages [ 5392.352579] LustreError: 165959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5392.376912] LustreError: 165959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5392.852335] Lustre: server umount lustre-MDT0000 complete [ 5396.215344] LustreError: 165937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140503 with bad export cookie 8046241673424516144 [ 5396.226764] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5396.230829] LustreError: 165937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5396.674688] Lustre: server umount lustre-MDT0001 complete [ 5404.990545] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:48:30 (1783140510) [ 5421.679664] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 5432.412069] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5464.435476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5464.634656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5464.667329] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5464.739795] Lustre: lustre-MDT0000: new disk, initializing [ 5464.803806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.823719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5468.863801] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5478.574122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5478.716922] Lustre: 172670:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5479.045325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5483.926734] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5493.271695] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5500.359180] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5507.055601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5507.480235] Lustre: lustre-OST0000: new disk, initializing [ 5507.482647] Lustre: Skipped 1 previous similar message [ 5507.485706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5507.492024] Lustre: Skipped 2 previous similar messages [ 5507.496434] Lustre: 174645:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5513.415896] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5515.838939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5515.846438] Lustre: Skipped 1 previous similar message [ 5515.849857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5515.943675] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5522.686147] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5531.619918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5531.621253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5531.634559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5531.650899] LustreError: Skipped 1 previous similar message [ 5535.611176] Lustre: server umount lustre-OST0000 complete [ 5540.320884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5544.968667] Lustre: server umount lustre-MDT0000 complete [ 5546.979495] LustreError: 175315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5546.988839] LustreError: 175315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5548.331100] LustreError: 174644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140655 with bad export cookie 8046241673424517026 [ 5548.335239] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5548.340055] LustreError: 174644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5548.678396] Lustre: server umount lustre-MDT0001 complete [ 5556.512173] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:51:02 (1783140662) [ 5570.757697] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 5581.394290] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5615.663171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5615.869398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5615.896779] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5615.969597] Lustre: lustre-MDT0000: new disk, initializing [ 5616.049131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5616.051551] Lustre: Skipped 2 previous similar messages [ 5616.077608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5620.033568] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5629.814143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5629.937812] Lustre: 178969:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5630.193967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5633.916970] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5642.546622] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5647.500728] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5653.466093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5653.814527] Lustre: 180944:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5659.879184] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5662.258335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5662.341504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5668.439582] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5677.537852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5677.539440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5677.548230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5677.548241] Lustre: Skipped 4 previous similar messages [ 5677.579615] Lustre: Skipped 4 previous similar messages [ 5689.311299] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5689.437247] Lustre: server umount lustre-OST0000 complete [ 5698.612370] Lustre: server umount lustre-MDT0000 complete [ 5702.257909] LustreError: 178962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140809 with bad export cookie 8046241673424518111 [ 5702.261453] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5702.271191] LustreError: 178962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5702.747462] Lustre: server umount lustre-MDT0001 complete [ 5712.859595] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5715.192811] Key type lgssc unregistered [ 5715.486549] LNet: 182801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5715.491354] LNetError: 182801:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5715.509073] LNet: Removed LNI 192.168.202.145@tcp [ 5716.221219] Key type .llcrypt unregistered [ 5716.223197] Key type ._llcrypt unregistered [ 5741.232731] Key type ._llcrypt registered [ 5741.234561] Key type .llcrypt registered [ 5741.328247] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5742.414951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5742.530954] alg: No test for adler32 (adler32-zlib) [ 5743.591817] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 5743.801260] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 5745.488662] Key type lgssc registered [ 5746.817098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5754.762842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5754.785886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5756.297890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5760.080864] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5765.602483] LustreError: 184219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5765.623623] LustreError: 184219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5767.526049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5767.843342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5772.335522] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5779.441047] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5784.973793] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5791.972983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5792.447670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5798.608752] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5800.652183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5806.241355] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5824.481205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5824.492391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5824.524927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5826.533730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5826.539020] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5826.543425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5829.715644] Lustre: server umount lustre-OST0000 complete [ 5834.721064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5834.732385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5834.758525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5834.761458] Lustre: Skipped 1 previous similar message [ 5836.772341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5836.773358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5839.198635] Lustre: server umount lustre-MDT0000 complete [ 5841.896278] LustreError: 184946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5841.923731] LustreError: 184946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5842.777147] LustreError: 184198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140950 with bad export cookie 3931608266005533528 [ 5842.782171] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5842.790209] LustreError: 184198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5843.112909] Lustre: server umount lustre-MDT0001 complete [ 5854.011127] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5856.587171] Key type lgssc unregistered [ 5856.805596] LNet: 187771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5856.813198] LNetError: 187771:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5856.831495] LNet: Removed LNI 192.168.202.145@tcp [ 5857.642838] Key type .llcrypt unregistered [ 5857.644961] Key type ._llcrypt unregistered [ 5880.412315] Key type ._llcrypt registered [ 5880.416242] Key type .llcrypt registered [ 5880.517132] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5881.499725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5881.563506] alg: No test for adler32 (adler32-zlib) [ 5882.577699] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 5882.761277] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 5884.407726] Key type lgssc registered [ 5885.550966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5891.899748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5891.912132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5893.222472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5897.492466] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5902.816913] LustreError: 189189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5902.824509] LustreError: 189189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5905.015663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5905.254222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5908.959452] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5915.373769] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5921.350606] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5928.677942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5929.155504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5936.494915] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5938.366251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5944.461568] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5952.249652] Lustre: lustre-OST0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 5954.019781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5954.025549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5954.045617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5954.055527] Lustre: Skipped 1 previous similar message [ 5957.357426] LustreError: 191767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5957.501409] Lustre: server umount lustre-OST0000 complete [ 5964.683979] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff913b4b1c7000.ost_server_uuid 50 [ 5965.931443] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5982.177669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5982.194969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.208753] Lustre: Skipped 1 previous similar message [ 5982.214470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5983.715559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.299196] Lustre: server umount lustre-MDT0000 complete [ 5988.825185] LustreError: 189170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141096 with bad export cookie 11578068596651192865 [ 5988.833471] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5989.215353] Lustre: server umount lustre-MDT0001 complete [ 5996.652397] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 00:58:22 (1783141102) [ 6002.123415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6002.534191] LustreError: 193106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6002.593699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6006.435418] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6007.789776] LustreError: 193107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6012.896721] LustreError: 193106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6013.876991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6014.175573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6018.237688] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6024.628849] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6029.896678] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6035.974914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6036.389101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6042.354465] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6047.672937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6049.380966] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6055.608038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6055.737471] Lustre: lustre-OST0001: new disk, initializing [ 6055.741258] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6055.749852] Lustre: 195946:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6055.843546] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6060.451581] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6063.166614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6063.174693] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6063.254585] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6068.291952] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6073.313947] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6073.320982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6073.346154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6073.351546] Lustre: Skipped 2 previous similar messages [ 6076.856764] Lustre: server umount lustre-OST0000 complete [ 6078.949768] LustreError: 196432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6082.348502] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6082.559644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6086.394811] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6086.618743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6099.429228] LustreError: 196431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.447399] LustreError: 196431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 6104.544915] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6104.546460] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6104.556034] LustreError: Skipped 1 previous similar message [ 6104.575696] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6104.592218] Lustre: Skipped 2 previous similar messages [ 6104.617479] Lustre: Skipped 3 previous similar messages [ 6106.250873] Lustre: server umount lustre-OST0001 complete [ 6114.787372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6114.793716] Lustre: Skipped 3 previous similar messages [ 6115.660698] Lustre: server umount lustre-MDT0000 complete [ 6119.454707] LustreError: 193086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141227 with bad export cookie 11578068596651193936 [ 6119.466175] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6119.470074] LustreError: 193086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6119.886119] Lustre: server umount lustre-MDT0001 complete [ 6143.048566] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:00:48 (1783141248) [ 6149.470177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6149.683374] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6149.707261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6149.915566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6154.406504] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6161.856088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6161.953085] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6161.976272] Lustre: 199382:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6165.482379] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6170.901586] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6174.596254] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6179.538556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6179.656779] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6184.179303] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6186.594400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6189.561468] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6194.710927] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6194.753220] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6199.492352] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6204.494782] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6206.947701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6206.956346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6206.968295] Lustre: Skipped 2 previous similar messages [ 6206.978470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6212.577813] Lustre: server umount lustre-OST0000 complete [ 6215.397395] LustreError: 201241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6215.409647] LustreError: 201241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6225.891613] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6225.895276] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6225.899800] LustreError: Skipped 1 previous similar message [ 6225.920177] Lustre: Skipped 1 previous similar message [ 6231.683865] Lustre: server umount lustre-OST0001 complete [ 6240.055145] Lustre: server umount lustre-MDT0000 complete [ 6242.073611] LustreError: 199373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141349 with bad export cookie 11578068596651195343 [ 6242.075038] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6242.079453] LustreError: 199373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6242.293829] Lustre: server umount lustre-MDT0001 complete [ 6257.271222] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:02:43 (1783141363) [ 6268.737339] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 6275.116464] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 6298.895451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6299.066775] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6299.087705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6299.156320] Lustre: lustre-MDT0000: new disk, initializing [ 6299.201378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6299.204966] Lustre: Skipped 3 previous similar messages [ 6299.211236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6301.998041] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6309.457104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6309.535533] Lustre: 207483:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6309.580995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6309.584502] Lustre: Skipped 1 previous similar message [ 6309.641751] Lustre: lustre-MDT0001: new disk, initializing [ 6309.745829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6309.757788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6313.622724] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6320.336500] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6324.179531] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6328.742293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6328.957928] Lustre: lustre-OST0000: new disk, initializing [ 6328.961300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6328.966477] Lustre: 209450:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6333.202975] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6336.549797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6336.563088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6336.641681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6339.968178] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6343.204440] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6343.329044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6346.591869] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6346.715543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6351.848266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6351.851567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6351.854648] LustreError: Skipped 2 previous similar messages [ 6351.856288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6351.856295] Lustre: Skipped 11 previous similar messages [ 6351.869928] Lustre: Skipped 4 previous similar messages [ 6354.424636] Lustre: server umount lustre-OST0000 complete [ 6365.325412] Lustre: server umount lustre-MDT0000 complete [ 6367.201144] LustreError: 207493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6367.224311] LustreError: 207493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 6371.217803] LustreError: 207474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141478 with bad export cookie 11578068596651196904 [ 6371.225375] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6371.236147] LustreError: 207474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6371.798827] Lustre: server umount lustre-MDT0001 complete [ 6384.314169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6391.997865] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6405.612240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.548550] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6419.608759] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6426.995621] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6434.865573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6435.112969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6435.116361] Lustre: Skipped 4 previous similar messages [ 6441.898970] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6452.011643] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6458.117373] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6458.454731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6463.254728] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6463.493042] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6466.028669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6466.040785] LustreError: Skipped 1 previous similar message [ 6466.043778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6466.060944] Lustre: Skipped 2 previous similar messages [ 6466.068538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6466.074428] Lustre: Skipped 4 previous similar messages [ 6471.671719] Lustre: server umount lustre-OST0000 complete [ 6477.534724] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6482.207408] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6512.430181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6517.933920] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6517.958495] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6518.709071] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6526.806697] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6529.806261] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6533.356717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.387569] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6538.734737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6540.731533] Lustre: server umount lustre-OST0000 complete [ 6550.717905] Lustre: server umount lustre-MDT0000 complete [ 6554.080694] LustreError: 211659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6554.116232] LustreError: 211659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6555.252724] LustreError: 211646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141662 with bad export cookie 11578068596651197786 [ 6555.268342] LustreError: 211646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6555.269288] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6555.668445] Lustre: server umount lustre-MDT0001 complete [ 6565.929192] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 01:07:51 (1783141671) [ 6572.838470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6578.296663] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6586.414049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6590.803781] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6598.426306] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6604.810668] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6611.490309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6618.886816] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6627.842801] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6633.406316] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6633.563354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6638.553402] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6638.787366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6645.606055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6645.823640] Lustre: lustre-OST0001: new disk, initializing [ 6645.835695] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6645.842753] Lustre: 221004:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6652.459257] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6655.626924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6655.640217] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6655.712968] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6662.329451] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6666.734666] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6667.018924] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6671.615590] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6671.871584] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6676.450727] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6676.451533] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6676.467028] LustreError: Skipped 2 previous similar messages [ 6676.497690] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6676.510774] Lustre: Skipped 6 previous similar messages [ 6676.525578] Lustre: Skipped 8 previous similar messages [ 6680.487443] Lustre: server umount lustre-OST0001 complete [ 6687.528673] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6687.791393] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6692.805956] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6692.984066] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6721.905878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6722.102258] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6722.107814] Lustre: Skipped 5 previous similar messages [ 6727.160825] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6727.199332] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6727.200338] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6728.586403] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6736.688376] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6739.038770] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6755.299613] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6755.364554] Lustre: server umount lustre-OST0001 complete [ 6778.375313] Lustre: server umount lustre-OST0000 complete [ 6788.640950] Lustre: server umount lustre-MDT0000 complete [ 6792.691240] LustreError: 217617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141900 with bad export cookie 11578068596651198752 [ 6792.694345] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6792.699272] LustreError: 217617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6799.372654] Lustre: server umount lustre-MDT0001 complete [ 6822.807847] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:12:08 (1783141928) [ 6829.112622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6829.468555] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6829.512312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6833.784139] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6841.846720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6841.930520] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6841.955467] Lustre: 226249:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6847.199419] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6854.987232] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6861.262963] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6867.715720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6867.919943] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6873.946669] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6881.682924] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6891.624517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6891.807413] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6898.161705] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6905.767876] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6907.887658] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6912.474408] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6912.667907] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6917.195878] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6917.475059] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6923.142363] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff913b4bed2800.ost_server_uuid 50 [ 6924.635439] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff913b4bed2800.ost_server_uuid in IDLE state after 0 sec [ 6926.031863] Lustre: Permanently deactivating lustre-OST0001 [ 6926.051699] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6930.003332] Lustre: Permanently reactivating lustre-OST0001 [ 6935.008074] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6935.013923] LustreError: Skipped 3 previous similar messages [ 6935.018625] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6935.029452] Lustre: Skipped 6 previous similar messages [ 6935.034036] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6935.037535] Lustre: Skipped 11 previous similar messages [ 6940.842189] Lustre: server umount lustre-OST0001 complete [ 6945.780618] LustreError: 230727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6945.811119] LustreError: 230727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [ 6959.071205] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6959.222811] Lustre: server umount lustre-OST0000 complete [ 6969.729681] Lustre: server umount lustre-MDT0000 complete [ 6973.308556] LustreError: 226241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142080 with bad export cookie 11578068596651200040 [ 6973.312633] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6973.315287] LustreError: 226241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6973.625595] Lustre: server umount lustre-MDT0001 complete [ 6995.102938] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:15:00 (1783142100) [ 7002.462867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7002.629727] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7002.647598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7002.656868] Lustre: Skipped 5 previous similar messages [ 7006.833150] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7015.082326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7015.154883] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7015.181553] Lustre: 232948:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7019.851784] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7030.088348] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7036.291586] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7042.871785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7043.078540] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7043.104570] Lustre: Permanently deactivating lustre-OST0000 [ 7043.117652] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7048.176044] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7048.715812] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7054.931833] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7060.754323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7060.858575] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7067.041552] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7068.895904] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7074.008461] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7075.884292] Lustre: Permanently reactivating lustre-OST0000 [ 7075.895874] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7075.902112] Lustre: Skipped 2 previous similar messages [ 7086.189190] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7086.189653] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 37 seconds. I think it's dead, and I am evicting it. exp 00000000801e2597, cur 1783142193 deadline 1783142186 last 1783142156 [ 7086.196563] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7086.219517] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7086.226461] Lustre: Skipped 1 previous similar message [ 7086.304915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7097.487244] Lustre: server umount lustre-OST0001 complete [ 7107.459810] Lustre: server umount lustre-OST0000 complete [ 7117.731224] Lustre: server umount lustre-MDT0000 complete [ 7121.584950] LustreError: 232941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142229 with bad export cookie 11578068596651201678 [ 7121.595654] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7121.598794] LustreError: 232941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7121.961068] Lustre: server umount lustre-MDT0001 complete [ 7133.344648] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 7135.870853] Key type lgssc unregistered [ 7136.133643] LNet: 237827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7136.140430] LNetError: 237827:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7137.189234] LNet: Removed LNI 192.168.202.145@tcp [ 7138.024101] Key type .llcrypt unregistered [ 7138.027864] Key type ._llcrypt unregistered [ 7162.006596] Key type ._llcrypt registered [ 7162.008587] Key type .llcrypt registered [ 7162.098861] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 7163.295992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7163.320668] alg: No test for adler32 (adler32-zlib) [ 7164.455993] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 7164.754914] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 7166.447186] Key type lgssc registered [ 7167.398843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7175.762558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7175.787028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7177.129613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7180.888103] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7186.415123] LustreError: 239297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7186.449921] LustreError: 239297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7189.168779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7189.291387] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7189.323683] Lustre: 239284:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7189.339413] Lustre: Permanently deactivating lustre-MDT0001 [ 7189.359777] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7189.602202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7192.790737] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7192.796352] LustreError: 239327:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7192.808258] LustreError: 239327:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7192.815948] LustreError: 239327:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7194.168125] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7200.740635] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7205.526797] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7211.932701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7212.142291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7215.204381] LustreError: 241135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7217.822743] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7220.705352] LustreError: 241137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7221.761230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7224.240889] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7226.851143] LustreError: 241136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7226.859411] LustreError: 241136:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7231.332914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7231.645547] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7237.114231] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7237.124392] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7237.885137] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7244.698555] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7248.506554] Lustre: Permanently reactivating lustre-MDT0001 [ 7248.520782] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7248.524938] Lustre: Skipped 2 previous similar messages [ 7250.416325] Lustre: lustre-MDT0001: Client ed57854f-a384-4c6d-a7d3-c96fcff7d668 (at 192.168.202.45@tcp) reconnecting [ 7256.306731] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7258.167871] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7264.774437] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7264.960086] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7268.827897] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7269.090347] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7271.744437] Lustre: Permanently deactivating lustre-MDT0001 [ 7271.759926] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7271.764213] Lustre: Skipped 2 previous similar messages [ 7280.296329] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7291.873812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7291.879890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7291.902281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7293.409451] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7293.410898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7293.418986] Lustre: Skipped 1 previous similar message [ 7293.429259] Lustre: Skipped 3 previous similar messages [ 7295.480971] Lustre: server umount lustre-MDT0000 complete [ 7298.535396] LustreError: 239293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7298.556247] LustreError: 239293:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7299.248563] LustreError: 239278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142406 with bad export cookie 9282732836115056573 [ 7299.264927] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7299.265869] LustreError: 239278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7299.598933] Lustre: server umount lustre-MDT0001 complete [ 7305.401481] Lustre: server umount lustre-OST0000 complete [ 7316.913360] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 01:20:22 (1783142422) [ 7327.525224] Lustre: server umount lustre-OST0001 complete [ 7333.892993] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 7344.063600] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 7381.999208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7382.203820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7382.212673] Lustre: Skipped 2 previous similar messages [ 7382.230924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7382.292984] Lustre: lustre-MDT0000: new disk, initializing [ 7382.363356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7382.377945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7388.204164] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7398.662743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7398.741719] Lustre: 247568:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7398.782671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7398.786044] Lustre: Skipped 1 previous similar message [ 7398.841972] Lustre: lustre-MDT0001: new disk, initializing [ 7398.923560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7398.967330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7398.979708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7403.390778] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7411.760785] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7416.609459] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7422.694988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7422.871821] Lustre: lustre-OST0000: new disk, initializing [ 7422.877196] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7422.884153] Lustre: 249539:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7422.964292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7428.881509] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7432.785228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7432.794317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7432.900461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7436.839693] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7438.322453] LustreError: 247576:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7445.694579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7445.841045] Lustre: lustre-OST0001: new disk, initializing [ 7445.847332] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7445.852146] Lustre: 250652:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7448.400728] LustreError: 247576:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7448.424749] LustreError: 250451:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7451.983727] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7454.822802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7454.833516] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7454.901135] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7458.455150] LustreError: 250451:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7460.102684] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7461.711648] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7465.445787] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7465.453472] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7465.472911] Lustre: Skipped 1 previous similar message [ 7465.481417] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7465.484493] Lustre: Skipped 1 previous similar message [ 7469.476497] Lustre: server umount lustre-OST0001 complete [ 7470.570727] LustreError: 249519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7470.583604] LustreError: 249519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7485.925500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7485.937173] Lustre: Skipped 1 previous similar message [ 7485.949140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7485.953970] Lustre: Skipped 2 previous similar messages [ 7491.048433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7491.049287] LustreError: 249523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7491.055310] Lustre: Skipped 1 previous similar message [ 7491.080772] LustreError: 249523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 7491.661383] Lustre: server umount lustre-OST0000 complete [ 7496.160499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7496.165824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7496.168606] LustreError: Skipped 1 previous similar message [ 7496.178606] Lustre: Skipped 1 previous similar message [ 7501.133748] Lustre: server umount lustre-MDT0000 complete [ 7504.329268] LustreError: 247561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142611 with bad export cookie 9282732836115059177 [ 7504.333773] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7504.339819] LustreError: 247561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7510.776446] Lustre: server umount lustre-MDT0001 complete [ 7521.389617] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 7523.660302] Key type lgssc unregistered [ 7523.924476] LNet: 252688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7523.929051] LNetError: 252688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7523.945634] LNet: Removed LNI 192.168.202.145@tcp [ 7524.593908] Key type .llcrypt unregistered [ 7524.598456] Key type ._llcrypt unregistered [ 7557.944792] Key type ._llcrypt registered [ 7557.948391] Key type .llcrypt registered [ 7558.048345] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 7559.064596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7559.123340] alg: No test for adler32 (adler32-zlib) [ 7560.180401] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 7560.454792] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 7562.183197] Key type lgssc registered [ 7563.032594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7569.832989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7569.849388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7571.117706] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7571.135982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7571.263277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7575.371185] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7582.514658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7582.589987] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7582.613603] Lustre: 254894:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7582.887026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7586.510136] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7593.477810] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7599.373445] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7606.092190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7606.287117] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7606.476410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7612.912416] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7613.113060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7620.722882] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7674.849598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7674.862225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7674.872452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7674.876237] Lustre: Skipped 1 previous similar message [ 7678.094264] Lustre: server umount lustre-OST0000 complete [ 7682.145780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7693.028490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7693.255052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7695.333216] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7695.351184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7695.351435] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7695.378756] Lustre: Skipped 1 previous similar message [ 7698.334524] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7705.363990] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7746.529942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7746.531613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7746.535909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7746.539779] LustreError: Skipped 2 previous similar messages [ 7746.565522] Lustre: Skipped 2 previous similar messages [ 7749.777479] Lustre: server umount lustre-OST0000 complete [ 7756.768441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7756.773557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7756.791400] Lustre: Skipped 1 previous similar message [ 7756.802105] Lustre: Skipped 1 previous similar message [ 7759.419392] Lustre: server umount lustre-MDT0000 complete [ 7762.538955] LustreError: 257595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142870 with bad export cookie 5886711437246907443 [ 7762.545920] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7762.554827] LustreError: 257595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7762.941832] Lustre: server umount lustre-MDT0001 complete [ 7770.920540] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 7772.937227] Key type lgssc unregistered [ 7773.151870] LNet: 260320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7773.163072] LNetError: 260320:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7773.180911] LNet: Removed LNI 192.168.202.145@tcp [ 7773.783329] Key type .llcrypt unregistered [ 7773.784965] Key type ._llcrypt unregistered [ 7791.445325] Key type ._llcrypt registered [ 7791.447208] Key type .llcrypt registered [ 7791.547733] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 7792.488789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7792.507164] alg: No test for adler32 (adler32-zlib) [ 7793.532291] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 7793.721253] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 7795.383176] Key type lgssc registered [ 7796.225311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7802.090307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7802.101157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7803.371995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7806.885414] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7813.612721] LustreError: 261715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7813.625776] LustreError: 261715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7813.740610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7813.978579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7817.325615] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7823.321324] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7828.239786] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7833.617456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7833.849258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7838.600452] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7844.377264] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7849.196617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7869.926785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7869.930478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7869.933893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7869.942836] Lustre: Skipped 1 previous similar message [ 7873.409713] Lustre: server umount lustre-OST0000 complete [ 7880.160893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.165160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7880.168974] Lustre: Skipped 1 previous similar message [ 7880.171789] Lustre: Skipped 1 previous similar message [ 7883.022717] Lustre: server umount lustre-MDT0000 complete [ 7885.291027] LustreError: 261711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7885.307407] LustreError: 261711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7885.930982] LustreError: 261697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142993 with bad export cookie 15481068384434694477 [ 7885.939501] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7886.196977] Lustre: server umount lustre-MDT0001 complete [ 7894.922717] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 7896.935529] Key type lgssc unregistered [ 7897.143515] LNet: 266019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7897.149047] LNetError: 266019:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7898.214379] LNet: Removed LNI 192.168.202.145@tcp [ 7899.041637] Key type .llcrypt unregistered [ 7899.045980] Key type ._llcrypt unregistered [ 7912.871090] Key type ._llcrypt registered [ 7912.873116] Key type .llcrypt registered [ 7912.969283] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 7913.733047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7913.739498] alg: No test for adler32 (adler32-zlib) [ 7914.686850] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 7914.884982] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 7916.519268] Key type lgssc registered [ 7917.290523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7922.975343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7923.000216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7924.272423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7927.446258] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7933.410808] LustreError: 267219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.434659] LustreError: 267219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7933.578128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7933.837871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7937.204721] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7942.867812] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7946.688539] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7951.076234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7951.299720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7955.263254] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7960.541483] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7968.747644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7984.099698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7984.100800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7984.125588] LustreError: Skipped 1 previous similar message [ 7984.128321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7985.260253] Lustre: server umount lustre-OST0000 complete [ 7989.220669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7989.227631] Lustre: Skipped 1 previous similar message [ 7989.231392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7989.236086] Lustre: Skipped 2 previous similar messages [ 7994.194461] Lustre: server umount lustre-MDT0000 complete [ 7994.337610] LustreError: 269482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7994.347199] LustreError: 269482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7996.695918] LustreError: 267199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143104 with bad export cookie 528379882872909241 [ 7996.708364] LustreError: 267199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7996.708768] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7996.952710] Lustre: server umount lustre-MDT0001 complete [ 8005.871913] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 8007.646792] Key type lgssc unregistered [ 8007.866771] LNet: 270914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8007.874261] LNetError: 270914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8007.893041] LNet: Removed LNI 192.168.202.145@tcp [ 8008.453522] Key type .llcrypt unregistered [ 8008.455104] Key type ._llcrypt unregistered [ 8025.219789] Key type ._llcrypt registered [ 8025.221884] Key type .llcrypt registered [ 8025.297882] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 8026.134231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8026.194104] alg: No test for adler32 (adler32-zlib) [ 8027.174251] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 8027.342769] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 8028.999159] Key type lgssc registered [ 8029.892106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8034.954498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8034.974654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8036.215307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8039.221922] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8045.362099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8045.541668] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8045.657445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8048.585865] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8053.721146] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8057.231678] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8061.449092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8061.701646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8065.624483] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8070.500662] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8081.197654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8101.858820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8101.860533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8101.883570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8101.895511] Lustre: Skipped 1 previous similar message [ 8101.912051] Lustre: Skipped 1 previous similar message [ 8106.979600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8106.983945] Lustre: Skipped 1 previous similar message [ 8107.241619] Lustre: server umount lustre-OST0000 complete [ 8112.097063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8112.097857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8112.101363] LustreError: Skipped 1 previous similar message [ 8112.102363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8112.106484] Lustre: Skipped 1 previous similar message [ 8112.117773] Lustre: Skipped 1 previous similar message [ 8115.635173] Lustre: server umount lustre-MDT0000 complete [ 8117.220026] LustreError: 274065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8117.235979] LustreError: 274065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8117.933272] LustreError: 272292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143225 with bad export cookie 9627582595133264647 [ 8117.940147] LustreError: 272292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8117.942264] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8118.184994] Lustre: server umount lustre-MDT0001 complete [ 8124.878403] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 8126.537395] Key type lgssc unregistered [ 8126.718975] LNet: 276637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8126.725861] LNetError: 276637:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8126.741183] LNet: Removed LNI 192.168.202.145@tcp [ 8127.214543] Key type .llcrypt unregistered [ 8127.215729] Key type ._llcrypt unregistered [ 8138.534713] Key type ._llcrypt registered [ 8138.538610] Key type .llcrypt registered [ 8138.618694] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 8139.153164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8139.165537] alg: No test for adler32 (adler32-zlib) [ 8140.134691] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 8140.324146] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 8141.978497] Key type lgssc registered [ 8142.780993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8147.956443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8147.983457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8149.289321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8152.464606] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8157.857320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8158.071539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8160.673906] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8165.565857] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8169.275249] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8174.246366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8174.475453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8179.340105] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8184.294409] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8193.977577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8203.239272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8203.246895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8203.263076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8204.257310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8204.263383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8204.265667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8204.278524] Lustre: Skipped 1 previous similar message [ 8208.128136] Lustre: server umount lustre-OST0000 complete [ 8214.496890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8214.503338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8214.511408] Lustre: Skipped 1 previous similar message [ 8214.521190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8217.238782] Lustre: server umount lustre-MDT0000 complete [ 8219.618945] LustreError: 277832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8219.628668] LustreError: 277832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8219.888838] LustreError: 277819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143327 with bad export cookie 2077576923990963341 [ 8219.898412] LustreError: 277819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8219.898883] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8220.151540] Lustre: server umount lustre-MDT0001 complete [ 8227.753149] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 8229.804571] Key type lgssc unregistered [ 8230.033946] LNet: 281566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8230.039442] LNetError: 281566:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8230.056250] LNet: Removed LNI 192.168.202.145@tcp [ 8230.669250] Key type .llcrypt unregistered [ 8230.676106] Key type ._llcrypt unregistered [ 8238.588992] Key type ._llcrypt registered [ 8238.592023] Key type .llcrypt registered [ 8238.683363] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8250.771143] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 8261.071406] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 8261.817021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8261.834426] alg: No test for adler32 (adler32-zlib) [ 8262.785303] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 8262.975054] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 8264.599235] Key type lgssc registered [ 8265.333814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8288.879644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8288.898670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8290.025173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8290.044140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8290.087969] Lustre: lustre-MDT0000: new disk, initializing [ 8290.124246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8290.135946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8292.583788] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8299.741322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8299.864684] Lustre: 285094:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8299.939967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8299.943972] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8299.944631] Lustre: Skipped 1 previous similar message [ 8300.009203] Lustre: lustre-MDT0001: new disk, initializing [ 8300.083225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8300.100187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8300.106351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8302.618664] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8309.548340] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8313.607164] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8318.291210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8318.491836] Lustre: lustre-OST0000: new disk, initializing [ 8318.494812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8318.501554] Lustre: 287063:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8318.554590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8322.608436] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8325.153041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8325.162614] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8325.231593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8329.159195] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8331.974710] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8332.109234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8334.900096] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8335.000633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8340.456664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.460826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8340.467667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8342.410967] Lustre: server umount lustre-OST0000 complete [ 8345.570357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8345.572225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8345.580854] Lustre: Skipped 2 previous similar messages [ 8345.593901] Lustre: Skipped 2 previous similar messages [ 8350.689174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8350.702174] Lustre: Skipped 1 previous similar message [ 8350.802739] Lustre: server umount lustre-MDT0000 complete [ 8352.890478] LustreError: 285087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143460 with bad export cookie 10619950716177744302 [ 8352.892826] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8352.896251] LustreError: 285087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8353.086131] Lustre: server umount lustre-MDT0001 complete [ 8357.773835] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 01:37:44 (1783143464) [ 8361.372204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8361.689438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8364.146453] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8367.074606] LustreError: 289473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8367.091144] LustreError: 289473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8368.790492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8369.018950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8371.470043] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8375.572916] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8378.482884] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8382.226995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8382.394431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8385.179783] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8388.787505] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8401.138893] Lustre: server umount lustre-OST0000 complete [ 8403.939937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8403.956974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8403.965808] Lustre: Skipped 1 previous similar message [ 8413.196520] Lustre: server umount lustre-MDT0000 complete [ 8414.176916] LustreError: 290891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8414.183576] LustreError: 290891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8415.166428] LustreError: 289452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143522 with bad export cookie 10619950716177745191 [ 8415.169509] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8415.179635] LustreError: 289452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8424.495505] Lustre: server umount lustre-MDT0001 complete [ 8430.464944] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 8432.267320] Key type lgssc unregistered [ 8432.454672] LNet: 293004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8432.458133] LNetError: 293004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8433.510962] LNet: Removed LNI 192.168.202.145@tcp [ 8434.028354] Key type .llcrypt unregistered [ 8434.030196] Key type ._llcrypt unregistered [ 8449.296238] Key type ._llcrypt registered [ 8449.297855] Key type .llcrypt registered [ 8449.357157] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 8458.086268] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 8458.647760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8458.720204] alg: No test for adler32 (adler32-zlib) [ 8459.697360] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 8459.847482] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 8461.447161] Key type lgssc registered [ 8462.170479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8484.967908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8484.983723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8486.154239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8486.174611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8486.240288] Lustre: lustre-MDT0000: new disk, initializing [ 8486.298230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8486.309957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8488.614555] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8494.801068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8494.867503] Lustre: 296801:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8494.902913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8494.905885] Lustre: Skipped 1 previous similar message [ 8494.958815] Lustre: lustre-MDT0001: new disk, initializing [ 8495.012427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8495.027978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8495.035771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8497.371855] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8502.967837] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8505.916073] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8509.533746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8509.726759] Lustre: lustre-OST2710: new disk, initializing [ 8509.729128] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8509.733315] Lustre: 298766:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8509.796954] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8512.348381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8512.355511] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8512.399891] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8513.214494] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8518.469678] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8522.051844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8522.150817] Lustre: lustre-OST03e8: new disk, initializing [ 8522.152579] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8522.156551] Lustre: 299829:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8522.213618] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8525.518441] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8528.914613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8528.922439] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8528.991465] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8529.831563] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8534.688559] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8534.811358] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8537.250179] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8537.373861] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8539.804137] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8539.919878] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8542.637218] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8542.770158] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8546.273355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8546.285062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8546.298448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8549.346410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8549.347213] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8549.354500] Lustre: Skipped 1 previous similar message [ 8549.363577] Lustre: Skipped 1 previous similar message [ 8551.680741] Lustre: server umount lustre-MDT0000 complete [ 8553.365436] LustreError: 296793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143661 with bad export cookie 10364921174196686925 [ 8553.368295] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8553.369710] LustreError: 296793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8553.544114] Lustre: server umount lustre-MDT0001 complete [ 8565.683390] Lustre: server umount lustre-OST2710 complete [ 8578.139579] Lustre: server umount lustre-OST03e8 complete [ 8589.099587] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 8595.578601] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 8615.100773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8615.253163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8615.268567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8615.328181] Lustre: lustre-MDT0000: new disk, initializing [ 8615.400527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8615.414053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8617.512068] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8623.480554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8623.534028] Lustre: 305073:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8623.561274] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8623.563561] Lustre: Skipped 1 previous similar message [ 8623.606818] Lustre: lustre-MDT0001: new disk, initializing [ 8623.662298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8623.684854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8623.692895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8625.969068] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8631.177854] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8633.772264] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8637.057648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8637.202715] Lustre: 307041:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8637.314788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8637.331993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 8640.144629] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8644.788399] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8647.281875] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8647.387088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8649.358159] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8649.455745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8652.769038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8652.770537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8652.771350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8652.771355] Lustre: Skipped 2 previous similar messages [ 8652.781992] Lustre: Skipped 2 previous similar messages [ 8656.490608] Lustre: server umount lustre-OST0000 complete [ 8659.424132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8659.428228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8659.433446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8659.436025] Lustre: Skipped 1 previous similar message [ 8664.437265] Lustre: server umount lustre-MDT0000 complete [ 8666.516730] LustreError: 305066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143774 with bad export cookie 10364921174196688598 [ 8666.518923] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8666.523484] LustreError: 305066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8666.714519] Lustre: server umount lustre-MDT0001 complete [ 8671.692569] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 01:42:57 (1783143777) [ 8672.515773] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8673.507157] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 01:42:59 (1783143779) [ 8675.357190] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 8689.623430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8689.738827] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8689.752287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8689.819313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8689.821747] Lustre: Skipped 1 previous similar message [ 8691.724872] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8695.840706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8695.883867] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8695.896398] Lustre: 310895:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8698.273420] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8715.556233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8715.660491] Lustre: Denying initial registration attempt from nid 192.168.202.145@tcp, specified as failover [ 8715.665321] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8715.669894] LustreError: 312346:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8715.673135] LustreError: 312346:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8715.675055] LustreError: 312346:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8715.714158] Lustre: server umount lustre-OST0000 complete [ 8715.717198] LustreError: 312346:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8719.328662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8719.329863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8719.337482] Lustre: Skipped 2 previous similar messages [ 8719.344586] Lustre: Skipped 3 previous similar messages [ 8729.568224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8729.570051] Lustre: Skipped 3 previous similar messages [ 8733.151151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8733.233639] Lustre: server umount lustre-MDT0000 complete [ 8734.787796] LustreError: 310888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143842 with bad export cookie 10364921174196689487 [ 8734.790371] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8734.792830] LustreError: 310888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8734.916420] Lustre: server umount lustre-MDT0001 complete [ 8739.572867] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 8741.035530] Key type lgssc unregistered [ 8741.201463] LNet: 313468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8741.205121] LNetError: 313468:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8741.215598] LNet: Removed LNI 192.168.202.145@tcp [ 8741.675174] Key type .llcrypt unregistered [ 8741.678050] Key type ._llcrypt unregistered [ 8747.664487] Key type ._llcrypt registered [ 8747.665685] Key type .llcrypt registered [ 8747.731054] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 8748.266861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8748.279927] alg: No test for adler32 (adler32-zlib) [ 8749.250215] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 8749.419535] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 8751.039492] Key type lgssc registered [ 8751.614780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8780.984334] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 8785.449786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8785.458441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8786.609692] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8786.621677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8786.707186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8788.747014] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8793.214162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8793.287891] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8793.310478] Lustre: 316207:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8793.449817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8795.336675] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8798.870543] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8801.447071] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8817.321912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8817.452100] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8817.526113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8819.979981] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8823.355159] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8824.459715] Lustre: server umount lustre-OST0000 complete [ 8826.360874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8826.361771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8826.365777] Lustre: Skipped 1 previous similar message [ 8831.457826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8831.460582] Lustre: Skipped 2 previous similar messages [ 8832.360786] Lustre: server umount lustre-MDT0000 complete [ 8834.094749] LustreError: 316200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143941 with bad export cookie 3257844476372781470 [ 8834.098901] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8834.099693] LustreError: 316200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8834.244971] Lustre: server umount lustre-MDT0001 complete [ 8838.649234] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 8839.777575] Key type lgssc unregistered [ 8839.909417] LNet: 319768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8839.912605] LNetError: 319768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8839.923893] LNet: Removed LNI 192.168.202.145@tcp [ 8840.288543] Key type .llcrypt unregistered [ 8840.290085] Key type ._llcrypt unregistered [ 8850.635491] Key type ._llcrypt registered [ 8850.637123] Key type .llcrypt registered [ 8850.687832] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 8851.050346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8851.083309] alg: No test for adler32 (adler32-zlib) [ 8851.963873] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 8852.083690] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 8853.679209] Key type lgssc registered [ 8854.118223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8857.256915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8857.268938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8858.489810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8860.162744] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8863.887628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8864.080727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8866.017580] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8869.541313] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8872.144463] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8877.091959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8877.274907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8879.788422] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8882.725896] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8893.921881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8893.923148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8893.928364] Lustre: Skipped 2 previous similar messages [ 8893.930405] Lustre: Skipped 1 previous similar message [ 8897.892351] Lustre: server umount lustre-MDT0000 complete [ 8899.042921] LustreError: 323652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8899.049339] LustreError: 323652:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8899.498240] LustreError: 321145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144007 with bad export cookie 15025860525116392858 [ 8899.499562] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8899.508511] LustreError: 321145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8904.162389] LustreError: 322402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8904.162812] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8904.180893] LustreError: 322402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8904.198331] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8904.201939] Lustre: Skipped 2 previous similar messages [ 8905.748439] Lustre: server umount lustre-MDT0001 complete [ 8909.038661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8912.823050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8924.575768] LustreError: 320474:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d5b2a095 x1869761959977472/t0(0) o250->MGC192.168.202.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8924.730396] LustreError: 322992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8924.738369] LustreError: 322992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8924.771609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8924.796769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:134 to 0x280000400:161) [ 8926.328250] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8928.802824] LustreError: 322993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8928.809773] LustreError: 322993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8929.934203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8930.018671] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.45@tcp (not set up) [ 8930.021947] Lustre: Skipped 1 previous similar message [ 8930.105986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8932.045754] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8934.180561] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8935.188214] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8935.981639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8938.730979] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8939.505336] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8944.609954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8944.611227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8944.616577] Lustre: Skipped 1 previous similar message [ 8947.503441] Lustre: server umount lustre-OST0000 complete [ 8949.728922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8949.730328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8949.734363] Lustre: Skipped 1 previous similar message [ 8949.744333] Lustre: Skipped 2 previous similar messages [ 8955.586491] Lustre: server umount lustre-MDT0000 complete [ 8957.387916] LustreError: 321144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144065 with bad export cookie 15025860525116400012 [ 8957.391405] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8957.393942] LustreError: 321144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8957.525335] Lustre: server umount lustre-MDT0001 complete [ 8962.421980] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 8963.888867] Key type lgssc unregistered [ 8964.043411] LNet: 327423:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8964.047150] LNetError: 327423:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8964.059380] LNet: Removed LNI 192.168.202.145@tcp [ 8964.507214] Key type .llcrypt unregistered [ 8964.508954] Key type ._llcrypt unregistered [ 8975.893691] Key type ._llcrypt registered [ 8975.895289] Key type .llcrypt registered [ 8975.958549] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 8976.603995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8976.618575] alg: No test for adler32 (adler32-zlib) [ 8977.541428] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 8977.656243] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 8979.247173] Key type lgssc registered [ 8979.747566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8982.956577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8982.964405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8984.192748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8985.980151] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8989.939110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8990.108246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8991.762611] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8995.060705] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8997.162077] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8999.393459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8999.394338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8999.400105] Lustre: Skipped 1 previous similar message [ 8999.404447] Lustre: Skipped 1 previous similar message [ 9000.416532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9000.423935] Lustre: Skipped 1 previous similar message [ 9004.719618] Lustre: server umount lustre-MDT0000 complete [ 9005.536875] LustreError: 328817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9005.548527] LustreError: 328817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9006.287382] LustreError: 328797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144113 with bad export cookie 16500511463082687670 [ 9006.290378] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9006.293232] LustreError: 328797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9006.418802] Lustre: server umount lustre-MDT0001 complete [ 9011.980806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9012.092368] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9012.108500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9012.185500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9013.811514] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9017.182228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9017.227050] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9017.241548] Lustre: 331402:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9017.266963] LustreError: 331412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9017.365337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9019.246549] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9022.596328] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9025.172624] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9030.761380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9030.880831] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9030.886279] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9030.890427] LustreError: 333287:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9030.894695] LustreError: 333287:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9030.898312] LustreError: 333287:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9030.936036] Lustre: server umount lustre-OST0000 complete [ 9030.937650] LustreError: 333287:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9037.160875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9037.235886] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9037.321697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9039.998511] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9043.016019] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9046.955658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9046.991762] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9047.050848] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 9047.053116] Lustre: lustre-OST0001: new disk, initializing [ 9047.058382] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9047.065722] Lustre: 334819:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9047.106418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:134 to 0x280000400:193) [ 9047.124516] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9049.637463] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9052.153109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9052.158483] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9052.182264] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9052.978264] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9057.250296] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9057.256855] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9057.257099] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9057.260368] Lustre: Skipped 1 previous similar message [ 9060.434783] Lustre: server umount lustre-OST0001 complete [ 9062.369590] LustreError: 334822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9062.370406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9062.371082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9062.371088] Lustre: Skipped 1 previous similar message [ 9062.372164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9062.376856] LustreError: 334822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9067.490974] LustreError: 333753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9067.495554] LustreError: 333753:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9068.254007] Lustre: server umount lustre-OST0000 complete [ 9072.608426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9072.610200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9072.614099] Lustre: Skipped 2 previous similar messages [ 9072.617957] Lustre: Skipped 4 previous similar messages [ 9084.383126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9084.489250] Lustre: server umount lustre-MDT0000 complete [ 9086.244207] LustreError: 331394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144193 with bad export cookie 16500511463082688209 [ 9086.249178] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9086.249321] LustreError: 331394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9086.380980] Lustre: server umount lustre-MDT0001 complete [ 9090.380573] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 9091.588139] Key type lgssc unregistered [ 9091.730412] LNet: 336778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9091.732690] LNetError: 336778:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9091.739419] LNet: Removed LNI 192.168.202.145@tcp [ 9092.079664] Key type .llcrypt unregistered [ 9092.080768] Key type ._llcrypt unregistered [ 9128.398052] Key type ._llcrypt registered [ 9128.399303] Key type .llcrypt registered [ 9128.452133] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 9135.147473] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 9135.592144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9135.607439] alg: No test for adler32 (adler32-zlib) [ 9136.486796] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 9136.595097] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 9138.183182] Key type lgssc registered [ 9138.613603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9151.572242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9151.580201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9152.674622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9152.693899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9152.760244] Lustre: lustre-MDT0000: new disk, initializing [ 9152.797436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9152.808122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9154.582866] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9159.159175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9159.220527] Lustre: 341680:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9159.251290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9159.254688] Lustre: Skipped 1 previous similar message [ 9159.293700] Lustre: lustre-MDT0001: new disk, initializing [ 9159.335551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9159.352488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9159.356644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9160.939762] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9164.956031] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9166.968529] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9169.615205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9169.735668] Lustre: lustre-OST0000: new disk, initializing [ 9169.737825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9169.741267] Lustre: 343649:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9169.775918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9171.795800] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9175.996793] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9177.791289] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9179.128500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9179.131492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9179.165376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9179.929077] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 9181.397879] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9181.481563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9184.224945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9184.226816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9184.227534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9184.227674] LustreError: Skipped 1 previous similar message [ 9184.230759] Lustre: Skipped 1 previous similar message [ 9188.379225] Lustre: server umount lustre-OST0000 complete [ 9190.368044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9190.370308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9190.374494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9190.376768] Lustre: Skipped 1 previous similar message [ 9194.464174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9194.464654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9199.584238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9199.586449] Lustre: Skipped 2 previous similar messages [ 9204.191435] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9204.258746] Lustre: server umount lustre-MDT0000 complete [ 9205.785486] LustreError: 342473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144313 with bad export cookie 17315707656781595724 [ 9205.788982] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9205.790031] LustreError: 342473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9205.933476] Lustre: server umount lustre-MDT0001 complete [ 9209.409244] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 01:51:55 (1783144315) [ 9212.160913] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 01:51:58 (1783144318) [ 9215.198205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9215.349927] LustreError: 346416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9215.355724] LustreError: 346416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9215.379374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9216.782140] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9219.834680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9219.965597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9221.292980] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9223.689039] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9225.478907] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9228.005391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9228.151443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9230.070320] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9232.646969] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9247.804129] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9248.348470] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9248.928976] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9249.517294] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9250.224333] Lustre: Failing over lustre-MDT0000 [ 9250.427497] Lustre: server umount lustre-MDT0000 complete [ 9250.786350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9250.786621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9250.791981] LustreError: 346435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9250.798882] LustreError: 346435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9254.632113] LustreError: 346411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9259.746697] LustreError: 346412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9259.752461] LustreError: 346412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9263.501188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9263.545753] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9263.639993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9263.667598] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9264.867309] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9265.097627] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9268.713706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9268.723210] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9268.739827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9270.571243] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9271.161659] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9273.056182] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9273.711723] Lustre: Failing over lustre-MDT0000 [ 9273.826205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9273.827040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9273.827745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9273.830994] Lustre: Skipped 4 previous similar messages [ 9273.894404] Lustre: server umount lustre-MDT0000 complete [ 9275.114225] LustreError: 349324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9275.121085] LustreError: 349324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9284.067331] LustreError: 346416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9284.078230] LustreError: 346416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9287.287210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9287.365701] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9287.545553] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9289.331741] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9290.467175] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9292.774710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9292.778129] Lustre: Skipped 2 previous similar messages [ 9292.785887] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9292.804510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9295.093511] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9295.836769] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9303.009042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9303.010304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9303.012676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9303.012682] Lustre: Skipped 2 previous similar messages [ 9303.020016] Lustre: Skipped 1 previous similar message [ 9304.475519] Lustre: server umount lustre-MDT0000 complete [ 9305.882151] LustreError: 350399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144413 with bad export cookie 17315707656781598405 [ 9305.885663] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9305.886513] LustreError: 350399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9306.042925] Lustre: server umount lustre-MDT0001 complete [ 9318.127064] Lustre: server umount lustre-OST0000 complete [ 9324.862251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9325.097850] LustreError: 352769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9325.104674] LustreError: 352769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9325.139874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9325.144587] Lustre: Skipped 1 previous similar message [ 9326.766178] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9330.134173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9331.767532] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9334.392676] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9336.370678] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9338.615370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9340.733406] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9343.210785] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9349.996581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9357.530719] Lustre: server umount lustre-OST0000 complete [ 9360.352667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9360.353195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9360.357275] Lustre: Skipped 2 previous similar messages [ 9360.358992] Lustre: Skipped 2 previous similar messages [ 9368.569316] Lustre: server umount lustre-MDT0000 complete [ 9369.935230] LustreError: 352750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144477 with bad export cookie 17315707656781599126 [ 9369.938775] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9369.940948] LustreError: 352750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9380.376784] Lustre: server umount lustre-MDT0001 complete [ 9384.203715] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 9385.289720] Key type lgssc unregistered [ 9385.421442] LNet: 356323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9385.424864] LNetError: 356323:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9385.438559] LNet: Removed LNI 192.168.202.145@tcp [ 9385.796863] Key type .llcrypt unregistered [ 9385.798162] Key type ._llcrypt unregistered [ 9396.188636] Key type ._llcrypt registered [ 9396.189881] Key type .llcrypt registered [ 9396.229675] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 9396.633884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9396.658182] alg: No test for adler32 (adler32-zlib) [ 9397.522987] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 9397.612457] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 9399.199147] Key type lgssc registered [ 9399.579358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9402.242206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9402.249154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9403.408570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9404.961953] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9407.985798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9408.108694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9409.451638] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9412.188981] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9414.060267] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9416.660322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9416.783805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9419.088562] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9421.790459] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9430.120983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9438.688679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9438.691925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9438.697958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9440.739216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9440.739232] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9440.742223] Lustre: Skipped 1 previous similar message [ 9440.746319] Lustre: Skipped 1 previous similar message [ 9442.592308] Lustre: server umount lustre-MDT0000 complete [ 9444.132392] LustreError: 357797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144551 with bad export cookie 18799925196746010 [ 9444.135783] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9444.137667] LustreError: 357797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9444.266940] Lustre: server umount lustre-MDT0001 complete [ 9455.976392] Lustre: server umount lustre-OST0000 complete [ 9460.831955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9460.998367] LustreError: 361542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9461.004989] LustreError: 361542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9461.028907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9462.507383] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9465.582170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9465.723515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9467.272502] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9469.874109] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9471.792857] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9474.221105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9474.352289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9476.361118] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9478.895528] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9485.609453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9496.543778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9496.545983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9496.550490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9501.152167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9501.154111] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9502.547018] Lustre: server umount lustre-MDT0000 complete [ 9503.761368] LustreError: 361522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144611 with bad export cookie 18799925196747109 [ 9503.764839] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9503.766378] LustreError: 361522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9503.874893] Lustre: server umount lustre-MDT0001 complete [ 9515.357626] Lustre: server umount lustre-OST0000 complete [ 9520.322638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9520.480327] LustreError: 365469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9520.485373] LustreError: 365469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9520.500776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9521.836471] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9524.921136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9526.610896] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9529.152124] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9530.971485] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9533.193617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9535.133798] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9537.507718] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9546.597916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9550.816563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9550.818810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9550.822291] Lustre: Skipped 1 previous similar message [ 9550.824426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9550.826387] Lustre: Skipped 2 previous similar messages [ 9555.279526] Lustre: server umount lustre-OST0000 complete [ 9562.080422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9562.082924] Lustre: Skipped 5 previous similar messages [ 9562.581831] Lustre: server umount lustre-MDT0000 complete [ 9563.798976] LustreError: 365450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144671 with bad export cookie 18799925196748117 [ 9563.802590] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9563.803251] LustreError: 365450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9563.915897] Lustre: server umount lustre-MDT0001 complete [ 9567.480379] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 9568.460796] Key type lgssc unregistered [ 9568.591302] LNet: 369020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9568.595238] LNetError: 369020:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9568.603230] LNet: Removed LNI 192.168.202.145@tcp [ 9568.900050] Key type .llcrypt unregistered [ 9568.901419] Key type ._llcrypt unregistered [ 9577.912655] Key type ._llcrypt registered [ 9577.913954] Key type .llcrypt registered [ 9577.949388] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 9578.305157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9578.375755] alg: No test for adler32 (adler32-zlib) [ 9579.242687] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 9579.334695] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 9580.919166] Key type lgssc registered [ 9581.257575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9583.592890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9583.598390] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9583.601819] LustreError: 370437:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9583.609984] LustreError: 370437:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9583.613383] LustreError: 370437:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9583.617063] LustreError: 370437:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9583.621326] LustreError: 370437:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9585.977070] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9585.979871] LustreError: 370834:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9585.987981] LustreError: 370834:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9585.990891] LustreError: 370834:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9585.993456] LustreError: 370834:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9585.997435] LustreError: 370834:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9592.434753] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 9593.496458] Key type lgssc unregistered [ 9593.629528] LNet: 371836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9593.632312] LNetError: 371836:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9593.643489] LNet: Removed LNI 192.168.202.145@tcp [ 9593.972764] Key type .llcrypt unregistered [ 9593.974044] Key type ._llcrypt unregistered [ 9599.946049] Key type ._llcrypt registered [ 9599.946849] Key type .llcrypt registered [ 9599.987866] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 9606.274426] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 9606.575705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9606.596061] alg: No test for adler32 (adler32-zlib) [ 9607.450472] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 9607.536140] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 9609.119203] Key type lgssc registered [ 9609.501037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9622.595076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9622.605821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9623.724335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9623.739139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9623.780463] Lustre: lustre-MDT0000: new disk, initializing [ 9623.813592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9623.822891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9625.647634] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9630.056784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9630.097927] Lustre: 374966:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9630.122390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9630.124024] Lustre: Skipped 1 previous similar message [ 9630.165358] Lustre: lustre-MDT0001: new disk, initializing [ 9630.199548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9630.212923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9630.218073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9631.684262] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9635.527478] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9637.640785] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9640.119952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9640.234606] Lustre: lustre-OST0000: new disk, initializing [ 9640.236793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9640.239842] Lustre: 376933:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9640.275745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9642.291781] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9646.728408] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9648.134476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9648.139215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9648.174559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9648.623472] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9648.723488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9650.376182] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9650.448481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9653.217578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9653.218562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9653.221409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9653.225050] Lustre: Skipped 1 previous similar message [ 9657.245696] Lustre: server umount lustre-OST0000 complete [ 9660.895929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9660.898130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9660.902834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9660.904824] Lustre: Skipped 1 previous similar message [ 9663.455972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9663.456542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9668.576153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9668.578899] Lustre: Skipped 2 previous similar messages [ 9672.672292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9672.674332] Lustre: Skipped 1 previous similar message [ 9673.183101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9673.245712] Lustre: server umount lustre-MDT0000 complete [ 9674.533211] LustreError: 374958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144782 with bad export cookie 10495552419367234526 [ 9674.536851] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9674.537170] LustreError: 374958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9674.660414] Lustre: server umount lustre-MDT0001 complete [ 9677.624898] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9678.259738] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 01:59:44 (1783144784) [ 9680.769765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9681.013450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9682.566674] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9685.866903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9686.037493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9687.590795] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9690.322267] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9692.270712] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9694.676828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9694.792347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9696.840875] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9699.375358] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9701.902109] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9701.969771] Lustre: lustre-OST0001: new disk, initializing [ 9701.971982] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9701.975176] Lustre: 382224:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9704.211751] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9704.968420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9704.972313] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9705.003686] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9708.094034] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9710.048449] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9710.048608] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9710.051067] LustreError: Skipped 1 previous similar message [ 9710.056263] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9710.060079] Lustre: Skipped 1 previous similar message [ 9716.099797] Lustre: server umount lustre-OST0001 complete [ 9717.219242] LustreError: 382689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9717.226640] LustreError: 382689:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9722.339461] LustreError: 381209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9722.345409] LustreError: 381209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9727.461034] LustreError: 381212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9727.467432] LustreError: 381212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9732.064852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9732.069775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9732.078112] Lustre: Skipped 1 previous similar message [ 9732.080757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9732.083617] Lustre: Skipped 3 previous similar messages [ 9732.579695] LustreError: 381212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9732.586538] LustreError: 381212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9734.429365] Lustre: server umount lustre-OST0000 complete [ 9737.183905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9737.187055] LustreError: Skipped 1 previous similar message [ 9742.109043] Lustre: server umount lustre-MDT0000 complete [ 9742.816224] LustreError: 379393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9742.822141] LustreError: 379393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9743.434309] LustreError: 379374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144851 with bad export cookie 10495552419367235394 [ 9743.437724] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9743.438274] LustreError: 379374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9749.618471] Lustre: server umount lustre-MDT0001 complete [ 9753.253052] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 9754.264639] Key type lgssc unregistered [ 9754.399319] LNet: 384219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9754.401618] LNetError: 384219:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9754.414401] LNet: Removed LNI 192.168.202.145@tcp [ 9754.746994] Key type .llcrypt unregistered [ 9754.748346] Key type ._llcrypt unregistered [ 9772.208435] Key type ._llcrypt registered [ 9772.209472] Key type .llcrypt registered [ 9772.247952] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 9772.700698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9772.710073] alg: No test for adler32 (adler32-zlib) [ 9773.566148] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 9773.658984] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 9775.255120] Key type lgssc registered [ 9775.608280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9778.157878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9778.171624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9779.257560] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9779.264871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9779.326254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9780.669087] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9783.642659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9783.679849] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9783.690853] Lustre: 386993:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9783.785476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9785.202356] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9787.548230] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9789.390966] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9792.480737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9792.481235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9792.484603] Lustre: Skipped 1 previous similar message [ 9796.181696] Lustre: server umount lustre-MDT0000 complete [ 9797.527089] LustreError: 386984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144905 with bad export cookie 13396982617303675843 [ 9797.529690] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9797.531771] LustreError: 386984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9797.651508] Lustre: server umount lustre-MDT0001 complete [ 9800.871636] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 02:01:47 (1783144907) [ 9803.129323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9803.306994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9804.639764] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9807.391744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9807.506071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9808.932863] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9811.471642] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9813.323086] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9815.646025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9815.712691] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9815.768238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9817.730697] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9820.255681] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9823.965318] Lustre: Permanently deactivating lustre-OST0000 [ 9823.970761] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9825.261988] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9825.863095] LustreError: 392338:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9825.866409] LustreError: 392338:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 9825.868641] LustreError: 392338:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9826.451479] LustreError: 392386:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 9826.454506] LustreError: 392386:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9833.176113] Lustre: server umount lustre-OST0000 complete [ 9834.072769] LustreError: 392635:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 9834.075035] LustreError: 392635:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9838.559879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9838.563246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9838.568453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9838.571321] Lustre: Skipped 1 previous similar message [ 9839.584099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9839.584586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9844.704143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9844.706547] Lustre: Skipped 2 previous similar messages [ 9849.311144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9849.371961] Lustre: server umount lustre-MDT0000 complete [ 9849.825132] LustreError: 389517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9849.832677] LustreError: 389517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9850.617081] LustreError: 389479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144958 with bad export cookie 13396982617303676410 [ 9850.620665] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9850.621718] LustreError: 389479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9861.140950] Lustre: server umount lustre-MDT0001 complete [ 9864.043597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9865.374155] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9866.306104] LustreError: 393792:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9868.493210] Lustre: 394130:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.145@tcp [ 9869.192428] Lustre: server umount MGS complete [ 9872.849277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9872.979777] LustreError: 394853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9872.984519] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9872.986955] Lustre: Skipped 1 previous similar message [ 9873.003181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9874.279899] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9877.035729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9877.124496] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9878.522208] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9880.985844] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9882.753711] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9883.395334] Lustre: Permanently reactivating lustre-OST0000 [ 9883.401463] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9883.404285] Lustre: Skipped 2 previous similar messages [ 9906.143122] LustreError: 396535:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9908.791834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9908.885713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9908.887618] Lustre: Skipped 1 previous similar message [ 9910.579394] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9912.708119] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9926.625736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9926.626815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9926.627944] LustreError: Skipped 1 previous similar message [ 9926.628418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9926.631637] Lustre: Skipped 1 previous similar message [ 9926.635652] Lustre: Skipped 1 previous similar message [ 9930.701175] Lustre: server umount lustre-OST0000 complete [ 9933.792046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9933.794603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9936.864022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9936.865974] Lustre: Skipped 2 previous similar messages [ 9938.193555] Lustre: server umount lustre-MDT0000 complete [ 9939.452252] LustreError: 394833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145047 with bad export cookie 13396982617303677446 [ 9939.455291] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9939.458052] LustreError: 394833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9939.574557] Lustre: server umount lustre-MDT0001 complete [ 9942.992599] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 9943.954952] Key type lgssc unregistered [ 9944.064386] LNet: 399430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9944.067212] LNetError: 399430:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9944.076312] LNet: Removed LNI 192.168.202.145@tcp [ 9944.372329] Key type .llcrypt unregistered [ 9944.373968] Key type ._llcrypt unregistered [ 9949.925735] Key type ._llcrypt registered [ 9949.926669] Key type .llcrypt registered [ 9949.961078] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 9955.974989] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 9956.265155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9956.273926] alg: No test for adler32 (adler32-zlib) [ 9957.131882] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 9957.215386] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 9958.791225] Key type lgssc registered [ 9959.156241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9969.855635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9969.864366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9970.937406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9970.946756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9970.981449] Lustre: lustre-MDT0000: new disk, initializing [ 9971.003295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9971.010820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9972.328025] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9976.311976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9976.351210] Lustre: 402556:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9976.370865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9976.372699] Lustre: Skipped 1 previous similar message [ 9976.404278] Lustre: lustre-MDT0001: new disk, initializing [ 9976.432037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9976.444563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9976.448225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9977.743418] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9981.198909] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9982.806587] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9984.802757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9984.897135] Lustre: lustre-OST0000: new disk, initializing [ 9984.898591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9984.902029] Lustre: 404525:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9984.930215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9986.699022] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9990.166364] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9991.577255] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9992.696458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9992.699892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9992.729777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9993.678708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 9994.935295] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9995.008471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9997.793103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9997.793421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9997.795849] LustreError: Skipped 1 previous similar message [ 9997.800335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9997.803212] Lustre: Skipped 1 previous similar message [10001.748683] Lustre: server umount lustre-OST0000 complete [10002.912984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10002.920232] Lustre: Skipped 1 previous similar message [10002.923528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10007.519731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10007.522460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10007.525876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10013.151995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10013.154018] Lustre: Skipped 3 previous similar messages [10017.247110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10017.301153] Lustre: server umount lustre-MDT0000 complete [10018.468695] LustreError: 402551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145126 with bad export cookie 1229460189828819927 [10018.469888] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10018.472538] LustreError: 402551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10018.586298] Lustre: server umount lustre-MDT0001 complete [10021.142903] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10021.678020] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 02:05:28 (1783145128) [10023.821130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10023.960085] LustreError: 406998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10023.966714] LustreError: 406998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10023.979478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10025.196798] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10028.035202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10028.160479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10029.594308] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10032.107866] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10033.917053] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10036.122175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10036.221502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10038.180071] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10040.336632] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10040.906555] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10051.871745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10053.599772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10053.601836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10053.606705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10059.276229] Lustre: server umount lustre-OST0000 complete [10064.351776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10064.353655] LustreError: Skipped 1 previous similar message [10064.354686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10064.357551] Lustre: Skipped 1 previous similar message [10064.358840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10064.360358] Lustre: Skipped 2 previous similar messages [10066.577423] Lustre: server umount lustre-MDT0000 complete [10067.676333] LustreError: 406979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145175 with bad export cookie 1229460189828820795 [10067.679046] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10067.679571] LustreError: 406979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10067.787955] Lustre: server umount lustre-MDT0001 complete [10071.053167] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [10072.007589] Key type lgssc unregistered [10072.128434] LNet: 410740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10072.130892] LNetError: 410740:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10072.141397] LNet: Removed LNI 192.168.202.145@tcp [10072.470686] Key type .llcrypt unregistered [10072.472406] Key type ._llcrypt unregistered [10080.135771] Key type ._llcrypt registered [10080.136695] Key type .llcrypt registered [10080.173322] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [10080.265326] Key type .llcrypt unregistered [10080.266251] Key type ._llcrypt unregistered [10086.754711] Key type ._llcrypt registered [10086.755767] Key type .llcrypt registered [10086.789976] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [10087.056388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10087.075136] alg: No test for adler32 (adler32-zlib) [10087.922126] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [10088.000685] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [10089.575150] Key type lgssc registered [10089.902790] Lustre: Echo OBD driver; http://www.lustre.org/ [10092.324934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10092.329445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10093.463706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10094.691414] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10097.249736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10097.348201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10099.032527] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10101.201264] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10102.753566] LustreError: 412925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10102.758633] LustreError: 412925:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10103.364560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10103.477666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10104.488096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10104.737422] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10108.896764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10108.899502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10108.905194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10109.920380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10109.922353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10109.925155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10113.615159] Lustre: server umount lustre-OST0000 complete [10115.040777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10115.044937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10115.046770] Lustre: Skipped 2 previous similar messages [10120.159872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10120.161249] Lustre: Skipped 1 previous similar message [10120.917113] Lustre: server umount lustre-MDT0000 complete [10122.081133] LustreError: 412906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145229 with bad export cookie 17689688914636670195 [10122.083624] LustreError: 412906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10122.083893] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10122.187853] Lustre: server umount lustre-MDT0001 complete [10125.459444] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [10126.378492] Key type lgssc unregistered [10126.501344] LNet: 416073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10126.503476] LNetError: 416073:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10126.511244] LNet: Removed LNI 192.168.202.145@tcp [10126.783785] Key type .llcrypt unregistered [10126.784625] Key type ._llcrypt unregistered [10134.822567] Key type ._llcrypt registered [10134.823514] Key type .llcrypt registered [10134.859891] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [10135.149703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10135.157709] alg: No test for adler32 (adler32-zlib) [10136.003132] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [10136.085851] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [10137.663150] Key type lgssc registered [10137.992507] Lustre: Echo OBD driver; http://www.lustre.org/ [10140.406127] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10140.415033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10155.871286] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10160.991210] LustreError: 417451:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [10187.615218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10189.469311] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10191.685591] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10193.672886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10193.824549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10193.843954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10195.077779] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10197.657878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10197.767396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10199.057209] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10201.261667] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10202.907837] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10208.226641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10208.226924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10208.230167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10208.232766] Lustre: Skipped 1 previous similar message [10211.146585] Lustre: server umount lustre-OST0000 complete [10213.344860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10213.344947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10213.350566] Lustre: Skipped 1 previous similar message [10213.352095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10213.353640] Lustre: Skipped 1 previous similar message [10218.391606] Lustre: server umount lustre-MDT0000 complete [10218.464037] LustreError: 420265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10218.468858] LustreError: 420265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10219.520034] LustreError: 418514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145327 with bad export cookie 16174936047198080817 [10219.522659] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10219.523474] LustreError: 418514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10219.629233] Lustre: server umount lustre-MDT0001 complete [10223.034148] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [10224.120751] Key type lgssc unregistered [10224.252444] LNet: 421016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10224.255244] LNetError: 421016:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10224.262315] LNet: Removed LNI 192.168.202.145@tcp [10224.587021] Key type .llcrypt unregistered [10224.588382] Key type ._llcrypt unregistered [10233.200513] Key type ._llcrypt registered [10233.202177] Key type .llcrypt registered [10233.235918] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [10233.547174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10233.577464] alg: No test for adler32 (adler32-zlib) [10234.425960] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [10234.499035] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [10236.079116] Key type lgssc registered [10236.405257] Lustre: Echo OBD driver; http://www.lustre.org/ [10238.697983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10238.704902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10239.832076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10241.061081] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10243.704178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10243.815743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10245.091616] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10247.316562] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10248.929019] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10251.022022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10251.143872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10252.960651] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10255.132912] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10264.485436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10269.666120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10269.666355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10269.673068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10269.674742] Lustre: Skipped 2 previous similar messages [10270.953021] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.45@tcp (stopping) [10272.726847] Lustre: server umount lustre-MDT0000 complete [10273.477515] LustreError: 422388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.45@tcp arrived at 1783145381 with bad export cookie 5477232357915223051 [10273.481723] LustreError: 422388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10274.115334] LustreError: 422390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145381 with bad export cookie 5477232357915222400 [10274.119952] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10274.120994] LustreError: 422390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10274.784478] LustreError: 422426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10274.785538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10274.785684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10274.789615] LustreError: 422426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10274.793503] Lustre: Skipped 2 previous similar messages [10274.799936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10279.903915] LustreError: 422407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10279.904286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10279.911693] LustreError: 422407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10280.208866] Lustre: server umount lustre-OST0000 complete [10282.491753] Lustre: server umount lustre-MDT0001 complete [10285.873284] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [10286.772510] Key type lgssc unregistered [10286.892318] LNet: 426111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10286.894347] LNetError: 426111:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10286.902284] LNet: Removed LNI 192.168.202.145@tcp [10287.177663] Key type .llcrypt unregistered [10287.178598] Key type ._llcrypt unregistered [10295.593357] Key type ._llcrypt registered [10295.594360] Key type .llcrypt registered [10295.632556] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [10296.068374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10296.083395] alg: No test for adler32 (adler32-zlib) [10296.951652] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [10297.043323] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [10298.623197] Key type lgssc registered [10298.975916] Lustre: Echo OBD driver; http://www.lustre.org/ [10301.539843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10301.545283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10302.683570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10304.054481] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10306.955869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10307.088114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10308.477216] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10310.833376] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10312.657492] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10314.881474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10314.991083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10316.866922] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10319.215729] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10328.295658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10332.639874] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10332.642368] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10332.647253] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10333.664488] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10333.667707] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10334.953218] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.45@tcp (stopping) [10334.954798] Lustre: Skipped 1 previous similar message [10336.843305] Lustre: server umount lustre-MDT0001 complete [10338.784146] LustreError: 427506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10338.784777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10338.784794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10338.788804] LustreError: 427506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10338.794072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10343.904143] LustreError: 428936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10343.904648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10344.402680] Lustre: server umount lustre-OST0000 complete [10345.729982] Lustre: server umount lustre-MDT0000 complete [10349.978509] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [10350.971250] Key type lgssc unregistered [10351.106354] LNet: 431208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10351.109289] LNetError: 431208:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10351.116225] LNet: Removed LNI 192.168.202.145@tcp [10351.420118] Key type .llcrypt unregistered [10351.421293] Key type ._llcrypt unregistered [10359.065583] Key type ._llcrypt registered [10359.066451] Key type .llcrypt registered [10359.101265] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [10364.722878] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [10364.990495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10365.001521] alg: No test for adler32 (adler32-zlib) [10365.850183] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [10365.931798] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [10367.511131] Key type lgssc registered [10367.837706] Lustre: Echo OBD driver; http://www.lustre.org/ [10378.805519] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [10383.850057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10383.856637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10384.938366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10384.949581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10384.989123] Lustre: lustre-MDT0000: new disk, initializing [10385.010370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10385.018686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10386.326827] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10391.711760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10391.800982] Lustre: lustre-OST0000: new disk, initializing [10391.802465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10391.803759] Lustre: Skipped 1 previous similar message [10391.805503] Lustre: 436213:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10391.826705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10393.576476] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10397.047554] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10398.717594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10398.720886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10398.729278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10408.379369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10408.430418] Lustre: 435192:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10408.461687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10408.499182] Lustre: lustre-MDT0001: new disk, initializing [10408.565994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10408.584809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10408.589227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10410.129356] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10413.829885] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10413.911537] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10415.253868] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10418.371626] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 3 sec [10422.753778] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10422.753811] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10422.758750] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10422.763015] Lustre: Skipped 1 previous similar message [10424.044494] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.45@tcp (stopping) [10425.999359] Lustre: server umount lustre-MDT0001 complete [10432.992849] LustreError: 435199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10432.998139] LustreError: 435199:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10438.116071] LustreError: 435198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10438.127772] LustreError: 435198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10439.647777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10439.650368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10439.653927] Lustre: Skipped 1 previous similar message [10439.656040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10443.232422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10443.232906] LustreError: 435204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10444.427797] Lustre: server umount lustre-OST0000 complete [10451.827680] Lustre: server umount lustre-MDT0000 complete [10456.188282] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [10457.132323] Key type lgssc unregistered [10457.249361] LNet: 439974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10457.251713] LNetError: 439974:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10457.260246] LNet: Removed LNI 192.168.202.145@tcp [10457.554384] Key type .llcrypt unregistered [10457.556065] Key type ._llcrypt unregistered [10469.043786] Key type ._llcrypt registered [10469.045257] Key type .llcrypt registered [10469.084989] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [10469.176322] Key type .llcrypt unregistered [10469.177353] Key type ._llcrypt unregistered [10476.095622] Key type ._llcrypt registered [10476.097445] Key type .llcrypt registered [10476.135899] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [10476.454953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10476.478053] alg: No test for adler32 (adler32-zlib) [10477.329371] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [10477.419331] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [10478.999098] Key type lgssc registered [10479.381418] Lustre: Echo OBD driver; http://www.lustre.org/ [10488.400973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10488.408499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10489.492905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10489.503227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10489.538093] Lustre: lustre-MDT0000: new disk, initializing [10489.559553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10489.567431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10490.942569] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10495.266249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10495.307350] Lustre: 443321:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10495.330826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10495.332547] Lustre: Skipped 1 previous similar message [10495.373632] Lustre: lustre-MDT0001: new disk, initializing [10495.405971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10495.419471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10495.423961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10496.851140] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10500.715920] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10502.581754] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10504.975955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10505.088816] Lustre: lustre-OST0000: new disk, initializing [10505.090752] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10505.093590] Lustre: 445288:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10505.124369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10506.558689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10506.561707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10506.601725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10507.071808] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10510.480560] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10515.935812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10515.938069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10515.941784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10516.961364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10516.961475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10516.964993] Lustre: Skipped 1 previous similar message [10516.966526] Lustre: Skipped 1 previous similar message [10518.745274] Lustre: server umount lustre-MDT0000 complete [10519.978220] LustreError: 444116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145627 with bad export cookie 14382114243133278053 [10519.981110] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10519.981979] LustreError: 444116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10520.081458] Lustre: server umount lustre-MDT0001 complete [10530.453484] Lustre: server umount lustre-OST0000 complete [10533.475853] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 02:14:00 (1783145640) [10535.877714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10536.024219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10537.207121] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10539.828890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10541.226749] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10543.400988] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10544.941590] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10559.969959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10560.040490] Lustre: Found index 0 for lustre-OST0000, updating log [10560.101407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10560.103912] Lustre: Skipped 1 previous similar message [10561.126781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10562.095695] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10564.226983] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10566.623823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10566.625744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10566.630087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10566.631540] Lustre: Skipped 1 previous similar message [10570.721615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10570.724803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10570.729754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10571.351283] Lustre: server umount lustre-OST0000 complete [10583.805820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10583.806711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10583.809880] Lustre: Skipped 1 previous similar message [10593.247935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10593.249439] Lustre: Skipped 6 previous similar messages [10598.367139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10598.425866] Lustre: server umount lustre-MDT0000 complete [10599.690209] LustreError: 447295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145707 with bad export cookie 14382114243133279432 [10599.693546] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10599.695151] LustreError: 447295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10599.814463] Lustre: server umount lustre-MDT0001 complete [10602.636657] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 02:15:09 (1783145709) [10626.911217] LNet: 1 local NIs in recovery (showing 1): 192.168.202.145@tcp [11182.139732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11182.279723] LustreError: 453711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11182.286746] LustreError: 453711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11182.301659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11183.609932] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11186.348345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11186.465639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11187.740651] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11190.044319] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11191.801444] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11207.010308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11207.114638] Lustre: Found index 0 for lustre-OST0000, updating log [11207.229608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11209.182268] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11209.321333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [11211.538025] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11213.216894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11213.221117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11213.223870] Lustre: Skipped 2 previous similar messages [11218.316685] Lustre: server umount lustre-OST0000 complete [11220.577549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11220.582099] Lustre: Skipped 1 previous similar message [11225.694445] Lustre: server umount lustre-MDT0000 complete [11225.699190] LustreError: 455155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11225.703725] LustreError: 455155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11226.721105] LustreError: 453712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11226.726616] LustreError: 453712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11226.975486] LustreError: 453739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146334 with bad export cookie 14382114243133280496 [11226.978709] LustreError: 453739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11226.979518] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11227.099785] Lustre: server umount lustre-MDT0001 complete [11240.287269] LNet: 1 local NIs in recovery (showing 1): 192.168.202.145@tcp [11244.288446] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [11247.339529] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [11258.998290] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 02:26:05 (1783146365) [11265.012084] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [11266.069265] Key type lgssc unregistered [11270.303420] LNet: 460357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11270.305881] LNetError: 460357:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11338.917320] LNet: Removed LNI 192.168.202.145@tcp [11339.254552] Key type .llcrypt unregistered [11339.255763] Key type ._llcrypt unregistered [11345.890747] Key type ._llcrypt registered [11345.891839] Key type .llcrypt registered [11345.939903] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [11346.360575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11346.370605] alg: No test for adler32 (adler32-zlib) [11347.256492] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [11347.369077] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [11348.959277] Key type lgssc registered [11349.380263] Lustre: Echo OBD driver; http://www.lustre.org/ [11350.996042] LNet: Added LNI 192.168.202.145@tcp100 [8/256/0/180] [11351.309609] LNet: Added LNI 192.168.202.145@tcp101 [8/256/0/180] [11351.621484] LNet: Added LNI 192.168.202.145@tcp102 [8/256/0/180] [11351.940471] LNet: Added LNI 192.168.202.145@tcp103 [8/256/0/180] [11352.249194] LNet: Added LNI 192.168.202.145@tcp104 [8/256/0/180] [11352.557889] LNet: Added LNI 192.168.202.145@tcp105 [8/256/0/180] [11352.852424] LNet: Added LNI 192.168.202.145@tcp106 [8/256/0/180] [11353.158305] LNet: Added LNI 192.168.202.145@tcp107 [8/256/0/180] [11353.455800] LNet: Added LNI 192.168.202.145@tcp108 [8/256/0/180] [11353.757307] LNet: Added LNI 192.168.202.145@tcp109 [8/256/0/180] [11354.059572] LNet: Added LNI 192.168.202.145@tcp110 [8/256/0/180] [11354.360300] LNet: Added LNI 192.168.202.145@tcp111 [8/256/0/180] [11354.670935] LNet: Added LNI 192.168.202.145@tcp112 [8/256/0/180] [11354.974511] LNet: Added LNI 192.168.202.145@tcp113 [8/256/0/180] [11355.287447] LNet: Added LNI 192.168.202.145@tcp114 [8/256/0/180] [11355.609159] LNet: Added LNI 192.168.202.145@tcp115 [8/256/0/180] [11355.908834] LNet: Added LNI 192.168.202.145@tcp116 [8/256/0/180] [11356.210642] LNet: Added LNI 192.168.202.145@tcp117 [8/256/0/180] [11356.507581] LNet: Added LNI 192.168.202.145@tcp118 [8/256/0/180] [11356.802774] LNet: Added LNI 192.168.202.145@tcp119 [8/256/0/180] [11357.093758] LNet: Added LNI 192.168.202.145@tcp120 [8/256/0/180] [11357.394290] LNet: Added LNI 192.168.202.145@tcp121 [8/256/0/180] [11357.673868] LNet: Added LNI 192.168.202.145@tcp122 [8/256/0/180] [11357.981520] LNet: Added LNI 192.168.202.145@tcp123 [8/256/0/180] [11358.283439] LNet: Added LNI 192.168.202.145@tcp124 [8/256/0/180] [11358.583938] LNet: Added LNI 192.168.202.145@tcp125 [8/256/0/180] [11358.888256] LNet: Added LNI 192.168.202.145@tcp126 [8/256/0/180] [11359.195959] LNet: Added LNI 192.168.202.145@tcp127 [8/256/0/180] [11359.482435] LNet: Added LNI 192.168.202.145@tcp128 [8/256/0/180] [11359.775600] LNet: Added LNI 192.168.202.145@tcp129 [8/256/0/180] [11360.077515] LNet: Added LNI 192.168.202.145@tcp130 [8/256/0/180] [11360.379104] LNet: Added LNI 192.168.202.145@tcp131 [8/256/0/180] [11360.694129] LNet: Added LNI 192.168.202.145@tcp132 [8/256/0/180] [11360.999620] LNet: Added LNI 192.168.202.145@tcp133 [8/256/0/180] [11361.299903] LNet: Added LNI 192.168.202.145@tcp134 [8/256/0/180] [11361.601106] LNet: Added LNI 192.168.202.145@tcp135 [8/256/0/180] [11363.760169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11363.766232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11364.801239] Lustre: 463355:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.145@tcp: can't add NID 192.168.202.145@tcp130: rc = -75 [11364.836699] LNetError: 460817:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.145 [11364.839774] LNetError: Protocol error connecting to 192.168.202.145@tcp119 at host 192.168.202.145:988: is it running a compatible version of Lustre? [11364.843724] LNetError: Skipped 6 previous similar messages [11364.888633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11364.901318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11364.937682] Lustre: lustre-MDT0000: new disk, initializing [11364.959439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11364.965106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11365.792400] LNet: 6 local NIs in recovery (showing 6): 192.168.202.145@tcp124, 192.168.202.145@tcp123, 192.168.202.145@tcp126, 192.168.202.145@tcp127, 192.168.202.145@tcp128, 192.168.202.145@tcp129 [11365.794776] LNetError: 463385:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.145 [11365.800430] LNetError: 463385:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [11365.809171] LNetError: Protocol error connecting to 192.168.202.145@tcp102 at host 192.168.202.145:988: is it running a compatible version of Lustre? [11365.814780] LNetError: Skipped 4 previous similar messages [11366.312437] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11367.843416] LNetError: 460819:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.145 [11367.846294] LNetError: 460819:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [11367.848671] LNetError: Protocol error connecting to 192.168.202.145@tcp128 at host 192.168.202.145:988: is it running a compatible version of Lustre? [11367.851377] LNetError: Skipped 1 previous similar message [11370.562798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11370.617783] Lustre: 463370:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11370.648517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11370.650310] Lustre: Skipped 1 previous similar message [11370.686969] Lustre: lustre-MDT0001: new disk, initializing [11370.701480] Lustre: 464177:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.145@tcp: can't add NID 192.168.202.145@tcp130: rc = -75 [11370.704941] Lustre: 464177:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11370.752343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11370.767241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11370.770839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11371.936410] LNetError: 463385:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.145 [11371.939263] LNetError: Protocol error connecting to 192.168.202.145@tcp128 at host 192.168.202.145:988: is it running a compatible version of Lustre? [11372.178677] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11374.454892] Lustre: 464614:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.145@tcp: can't add NID 192.168.202.145@tcp130: rc = -75 [11374.459613] Lustre: 464614:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11375.927354] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11377.697688] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11379.105117] LNetError: 460820:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.145 [11379.108565] LNetError: Protocol error connecting to 192.168.202.145@tcp128 at host 192.168.202.145:988: is it running a compatible version of Lustre? [11379.953296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11380.081560] Lustre: lustre-OST0000: new disk, initializing [11380.083238] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11380.085965] Lustre: 465348:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11380.123946] Lustre: 465353:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.145@tcp: can't add NID 192.168.202.145@tcp130: rc = -75 [11380.128815] Lustre: 465353:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [11380.152867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11380.277192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11380.284452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11380.335137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11382.117166] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11385.815920] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11390.222549] Lustre: server umount lustre-OST0000 complete [11395.488228] LNetError: 460817:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.145 [11395.491527] LNetError: Protocol error connecting to 192.168.202.145@tcp128 at host 192.168.202.145:988: is it running a compatible version of Lustre? [11396.065661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11396.066300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11396.069069] Lustre: Skipped 1 previous similar message [11396.072523] Lustre: Skipped 1 previous similar message [11401.184210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11401.186302] Lustre: Skipped 1 previous similar message [11402.237408] Lustre: server umount lustre-MDT0000 complete [11403.536598] LustreError: 463365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146511 with bad export cookie 3920160377376283440 [11403.539709] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11403.540467] LustreError: 463365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11411.872335] LNetError: 460817:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.145 [11411.875962] LNetError: 460817:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [11411.878090] LNetError: Protocol error connecting to 192.168.202.145@tcp126 at host 192.168.202.145:988: is it running a compatible version of Lustre? [11411.880979] LNetError: Skipped 10 previous similar messages [11414.029870] Lustre: server umount lustre-MDT0001 complete [11417.384601] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [11418.406613] Key type lgssc unregistered [11418.550293] LNet: 467260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11418.553720] LNetError: 467260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11418.556252] LNet: Removed LNI 192.168.202.145@tcp [11419.615134] LNet: Removed LNI 192.168.202.145@tcp100 [11419.616325] LNetError: 467260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11419.618565] LNetError: 467260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11419.622204] LNet: Removed LNI 192.168.202.145@tcp101 [11420.639152] LNet: Removed LNI 192.168.202.145@tcp102 [11420.640685] LNetError: 467260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11420.642627] LNetError: 467260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11420.644907] LNet: Removed LNI 192.168.202.145@tcp103 [11421.663141] LNet: Removed LNI 192.168.202.145@tcp104 [11422.687127] LNet: Removed LNI 192.168.202.145@tcp105 [11422.688704] LNetError: 467260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11422.691288] LNetError: 467260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11422.694067] LNet: Removed LNI 192.168.202.145@tcp106 [11423.711107] LNet: Removed LNI 192.168.202.145@tcp107 [11424.735200] LNet: Removed LNI 192.168.202.145@tcp108 [11425.759305] LNet: Removed LNI 192.168.202.145@tcp109 [11426.783252] LNet: Removed LNI 192.168.202.145@tcp110 [11426.784862] LNetError: 467260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11426.786973] LNetError: 467260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11426.789105] LNet: Removed LNI 192.168.202.145@tcp111 [11427.807223] LNet: Removed LNI 192.168.202.145@tcp112 [11428.831257] LNet: Removed LNI 192.168.202.145@tcp113 [11429.855149] LNet: Removed LNI 192.168.202.145@tcp114 [11430.879240] LNet: Removed LNI 192.168.202.145@tcp115 [11431.903277] LNet: Removed LNI 192.168.202.145@tcp116 [11432.927182] LNet: Removed LNI 192.168.202.145@tcp117 [11433.951077] LNet: Removed LNI 192.168.202.145@tcp118 [11434.975281] LNet: Removed LNI 192.168.202.145@tcp119 [11434.977227] LNetError: 467260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11434.980012] LNetError: 467260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11434.982200] LNet: Removed LNI 192.168.202.145@tcp120 [11435.999106] LNet: Removed LNI 192.168.202.145@tcp121 [11437.023216] LNet: Removed LNI 192.168.202.145@tcp122 [11438.047199] LNet: Removed LNI 192.168.202.145@tcp123 [11439.071158] LNet: Removed LNI 192.168.202.145@tcp124 [11440.095196] LNet: Removed LNI 192.168.202.145@tcp125 [11441.119177] LNet: Removed LNI 192.168.202.145@tcp126 [11442.143205] LNet: Removed LNI 192.168.202.145@tcp127 [11442.144576] LNet: Removed LNI 192.168.202.145@tcp128 [11443.167157] LNet: Removed LNI 192.168.202.145@tcp129 [11443.168484] LNet: Removed LNI 192.168.202.145@tcp130 [11443.169647] LNet: Removed LNI 192.168.202.145@tcp131 [11443.170767] LNet: Removed LNI 192.168.202.145@tcp132 [11443.171822] LNet: Removed LNI 192.168.202.145@tcp133 [11443.173215] LNet: Removed LNI 192.168.202.145@tcp134 [11443.180767] LNet: Removed LNI 192.168.202.145@tcp135 [11443.509711] Key type .llcrypt unregistered [11443.511081] Key type ._llcrypt unregistered [11451.266042] Key type ._llcrypt registered [11451.267850] Key type .llcrypt registered [11451.307315] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [11457.375138] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [11457.732857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11457.781683] alg: No test for adler32 (adler32-zlib) [11458.641361] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [11458.740948] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [11460.327165] Key type lgssc registered [11460.688708] Lustre: Echo OBD driver; http://www.lustre.org/ [11471.665618] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 02:29:38 (1783146578) [11477.421292] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [11478.408432] Key type lgssc unregistered [11478.540427] LNet: 471319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11478.543107] LNetError: 471319:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11478.555409] LNet: Removed LNI 192.168.202.145@tcp [11478.857266] Key type .llcrypt unregistered [11478.858349] Key type ._llcrypt unregistered [11485.466599] Key type ._llcrypt registered [11485.467469] Key type .llcrypt registered [11485.502975] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [11485.912654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11485.926109] alg: No test for adler32 (adler32-zlib) [11486.773933] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [11486.867144] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [11488.447135] Key type lgssc registered [11488.806407] Lustre: Echo OBD driver; http://www.lustre.org/ [11491.717902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11491.722786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11492.804257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11492.814269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11492.854044] Lustre: lustre-MDT0000: new disk, initializing [11492.876370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11492.882213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11494.203544] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11498.208545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11498.250274] Lustre: 472507:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11498.286435] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11498.288193] Lustre: Skipped 1 previous similar message [11498.323134] Lustre: lustre-MDT0001: new disk, initializing [11498.353232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11498.365536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11498.368947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11499.701986] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11503.324278] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11505.077408] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11507.202850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11507.300897] Lustre: lustre-OST0000: new disk, initializing [11507.302472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11507.304728] Lustre: 474476:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11507.329871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11507.562506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11507.566273] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11507.595062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11509.163949] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11512.745354] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11513.913567] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing lnet_if_list [11515.516091] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11515.525384] LNet: Added LNI 192.168.202.145@tcp10 [8/256/0/180] [11517.244433] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11517.254475] LNet: Added LNI 192.168.202.145@tcp11 [8/256/0/180] [11518.510556] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11518.520291] LNet: Added LNI 192.168.202.145@tcp12 [8/256/0/180] [11519.786139] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11519.795672] LNet: Added LNI 192.168.202.145@tcp13 [8/256/0/180] [11521.100812] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11521.111673] LNet: Added LNI 192.168.202.145@tcp14 [8/256/0/180] [11522.432325] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11522.441040] LNet: Added LNI 192.168.202.145@tcp15 [8/256/0/180] [11523.814209] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11523.823432] LNet: Added LNI 192.168.202.145@tcp16 [8/256/0/180] [11525.386819] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11525.397552] LNet: Added LNI 192.168.202.145@tcp17 [8/256/0/180] [11527.037583] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11527.048596] LNet: Added LNI 192.168.202.145@tcp18 [8/256/0/180] [11528.638987] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11528.648502] LNet: Added LNI 192.168.202.145@tcp19 [8/256/0/180] [11530.165412] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11530.176669] LNet: Added LNI 192.168.202.145@tcp20 [8/256/0/180] [11531.690359] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11531.699747] LNet: Added LNI 192.168.202.145@tcp21 [8/256/0/180] [11533.295215] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11533.304930] LNet: Added LNI 192.168.202.145@tcp22 [8/256/0/180] [11534.960450] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11534.972975] LNet: Added LNI 192.168.202.145@tcp23 [8/256/0/180] [11536.714852] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11536.727726] LNet: Added LNI 192.168.202.145@tcp24 [8/256/0/180] [11538.486217] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11538.495391] LNet: Added LNI 192.168.202.145@tcp25 [8/256/0/180] [11540.282890] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11540.293469] LNet: Added LNI 192.168.202.145@tcp26 [8/256/0/180] [11542.179502] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11542.189111] LNet: Added LNI 192.168.202.145@tcp27 [8/256/0/180] [11543.924391] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11543.934577] LNet: Added LNI 192.168.202.145@tcp28 [8/256/0/180] [11545.852707] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11545.862571] LNet: Added LNI 192.168.202.145@tcp29 [8/256/0/180] [11547.960918] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11547.974810] LNet: Added LNI 192.168.202.145@tcp30 [8/256/0/180] [11550.282656] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11550.297990] LNet: Added LNI 192.168.202.145@tcp31 [8/256/0/180] [11552.505706] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11552.517374] LNet: Added LNI 192.168.202.145@tcp32 [8/256/0/180] [11554.778387] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11554.794747] LNet: Added LNI 192.168.202.145@tcp33 [8/256/0/180] [11557.003385] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11557.017374] LNet: Added LNI 192.168.202.145@tcp34 [8/256/0/180] [11559.128315] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11559.141276] LNet: Added LNI 192.168.202.145@tcp35 [8/256/0/180] [11561.332618] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11561.347300] LNet: Added LNI 192.168.202.145@tcp36 [8/256/0/180] [11563.670530] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11563.683581] LNet: Added LNI 192.168.202.145@tcp37 [8/256/0/180] [11565.680259] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11565.691176] LNet: Added LNI 192.168.202.145@tcp38 [8/256/0/180] [11567.775274] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11567.784205] LNet: Added LNI 192.168.202.145@tcp39 [8/256/0/180] [11569.766615] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11569.779442] LNet: Added LNI 192.168.202.145@tcp40 [8/256/0/180] [11571.912951] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11571.924413] LNet: Added LNI 192.168.202.145@tcp41 [8/256/0/180] [11574.190252] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11574.199260] LNet: Added LNI 192.168.202.145@tcp42 [8/256/0/180] [11576.317261] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11576.328657] LNet: Added LNI 192.168.202.145@tcp43 [8/256/0/180] [11578.422807] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11578.433048] LNet: Added LNI 192.168.202.145@tcp44 [8/256/0/180] [11580.812686] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11580.821775] LNet: Added LNI 192.168.202.145@tcp45 [8/256/0/180] [11582.950096] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11582.958836] LNet: Added LNI 192.168.202.145@tcp46 [8/256/0/180] [11585.432063] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11585.442874] LNet: Added LNI 192.168.202.145@tcp47 [8/256/0/180] [11588.005700] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11588.020372] LNet: Added LNI 192.168.202.145@tcp48 [8/256/0/180] [11590.644613] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11590.656964] LNet: Added LNI 192.168.202.145@tcp49 [8/256/0/180] [11593.687531] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11593.700373] LNet: Added LNI 192.168.202.145@tcp50 [8/256/0/180] [11604.961743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11604.963912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11604.965845] Lustre: Skipped 1 previous similar message [11604.971410] Lustre: Skipped 2 previous similar messages [11607.703870] Lustre: server umount lustre-MDT0000 complete [11608.985459] LustreError: 472499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146716 with bad export cookie 11249978227194053742 [11608.990713] LustreError: 472499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11608.990774] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11609.118447] Lustre: server umount lustre-MDT0001 complete [11616.943302] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [11619.839393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11625.375141] Lustre: 471781:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783146717/real 1783146717] req@00000000b5f43fac x1869764723030400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783146733 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11625.386698] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11625.390888] Lustre: Skipped 1 previous similar message [11628.447125] Lustre: 471782:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783146720/real 1783146720] req@0000000048cb22f1 x1869764723030528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783146736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11630.559104] Lustre: 471783:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783146722/real 1783146722] req@00000000744bdc24 x1869764723030656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783146738 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11633.567353] LustreError: 471780:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000048cb22f1 x1869764723031424/t0(0) o250->MGC192.168.202.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11633.665343] LustreError: 474469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11633.671157] LustreError: 474469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11633.683226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11634.655069] Lustre: 471784:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783146726/real 1783146726] req@000000009880cb91 x1869764723030912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783146742 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11635.128161] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11635.745336] LustreError: 498659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11635.749557] LustreError: 498659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11638.104467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11638.245715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11639.737438] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11641.311092] Lustre: 471781:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783146732/real 1783146732] req@0000000003d7e039 x1869764723031168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783146748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11642.325073] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11643.365221] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11644.225851] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11644.388296] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11653.600540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11653.604591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11653.609665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11654.111882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11654.626319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11654.628507] Lustre: Skipped 1 previous similar message [11657.808116] Lustre: server umount lustre-OST0000 complete [11659.744897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11659.745702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11659.748193] Lustre: Skipped 2 previous similar messages [11659.750829] Lustre: Skipped 1 previous similar message [11664.864788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11664.868375] Lustre: Skipped 1 previous similar message [11665.178186] Lustre: server umount lustre-MDT0000 complete [11666.431279] LustreError: 472500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146774 with bad export cookie 11249978227194055758 [11666.434089] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11666.434716] LustreError: 472500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11666.555877] Lustre: server umount lustre-MDT0001 complete [11670.391779] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [11671.413510] Key type lgssc unregistered [11671.581904] LNet: 501436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11671.585918] LNetError: 501436:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11671.588471] LNet: Removed LNI 192.168.202.145@tcp [11671.590064] LNet: Removed LNI 192.168.202.145@tcp10 [11671.591337] LNet: Removed LNI 192.168.202.145@tcp11 [11671.592537] LNet: Removed LNI 192.168.202.145@tcp12 [11671.593808] LNet: Removed LNI 192.168.202.145@tcp13 [11671.595168] LNet: Removed LNI 192.168.202.145@tcp14 [11671.596358] LNet: Removed LNI 192.168.202.145@tcp15 [11671.597497] LNet: Removed LNI 192.168.202.145@tcp16 [11671.598567] LNet: Removed LNI 192.168.202.145@tcp17 [11671.599648] LNet: Removed LNI 192.168.202.145@tcp18 [11671.600709] LNet: Removed LNI 192.168.202.145@tcp19 [11671.601797] LNet: Removed LNI 192.168.202.145@tcp20 [11671.603530] LNet: Removed LNI 192.168.202.145@tcp21 [11671.605235] LNet: Removed LNI 192.168.202.145@tcp22 [11671.607074] LNet: Removed LNI 192.168.202.145@tcp23 [11671.608854] LNet: Removed LNI 192.168.202.145@tcp24 [11671.610566] LNet: Removed LNI 192.168.202.145@tcp25 [11671.612500] LNet: Removed LNI 192.168.202.145@tcp26 [11671.614681] LNet: Removed LNI 192.168.202.145@tcp27 [11671.616484] LNet: Removed LNI 192.168.202.145@tcp28 [11671.618227] LNet: Removed LNI 192.168.202.145@tcp29 [11671.619859] LNet: Removed LNI 192.168.202.145@tcp30 [11671.621790] LNet: Removed LNI 192.168.202.145@tcp31 [11671.623496] LNet: Removed LNI 192.168.202.145@tcp32 [11671.625181] LNet: Removed LNI 192.168.202.145@tcp33 [11671.626768] LNet: Removed LNI 192.168.202.145@tcp34 [11671.628343] LNet: Removed LNI 192.168.202.145@tcp35 [11671.629885] LNet: Removed LNI 192.168.202.145@tcp36 [11671.631481] LNet: Removed LNI 192.168.202.145@tcp37 [11671.633215] LNet: Removed LNI 192.168.202.145@tcp38 [11671.634760] LNet: Removed LNI 192.168.202.145@tcp39 [11671.636413] LNet: Removed LNI 192.168.202.145@tcp40 [11671.637935] LNet: Removed LNI 192.168.202.145@tcp41 [11671.639509] LNet: Removed LNI 192.168.202.145@tcp42 [11671.641167] LNet: Removed LNI 192.168.202.145@tcp43 [11671.642639] LNet: Removed LNI 192.168.202.145@tcp44 [11671.644202] LNet: Removed LNI 192.168.202.145@tcp45 [11671.645788] LNet: Removed LNI 192.168.202.145@tcp46 [11671.647468] LNet: Removed LNI 192.168.202.145@tcp47 [11671.649079] LNet: Removed LNI 192.168.202.145@tcp48 [11671.650636] LNet: Removed LNI 192.168.202.145@tcp49 [11671.658508] LNet: Removed LNI 192.168.202.145@tcp50 [11671.944508] Key type .llcrypt unregistered [11671.945486] Key type ._llcrypt unregistered [11678.535862] Key type ._llcrypt registered [11678.536858] Key type .llcrypt registered [11678.571481] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [11684.645072] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [11684.959777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11684.985098] alg: No test for adler32 (adler32-zlib) [11685.837145] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [11685.925387] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [11687.503197] Key type lgssc registered [11687.841459] Lustre: Echo OBD driver; http://www.lustre.org/ [11699.190044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11699.195997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11700.284854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11700.298047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11700.338324] Lustre: lustre-MDT0000: new disk, initializing [11700.363246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11700.370667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11701.731458] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11705.867215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11705.909122] Lustre: 504607:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11705.933777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11705.935549] Lustre: Skipped 1 previous similar message [11705.973511] Lustre: lustre-MDT0001: new disk, initializing [11706.007322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11706.021327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11706.027287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11707.333620] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11710.995517] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11712.807747] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11715.622614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11715.695263] Lustre: *** cfs_fail_loc=90f, val=0*** [11715.719896] Lustre: lustre-OST0000: new disk, initializing [11715.721386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11715.723635] Lustre: 506620:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11715.760743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11715.801231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11715.806187] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11715.814356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [11717.658212] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11721.372214] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11726.306614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11726.306902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11726.309042] LustreError: Skipped 1 previous similar message [11726.314100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11726.316463] Lustre: Skipped 1 previous similar message [11729.947704] Lustre: server umount lustre-OST0000 complete [11731.424496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11731.425089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11731.428075] Lustre: Skipped 2 previous similar messages [11736.544780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11736.546640] Lustre: Skipped 2 previous similar messages [11737.372064] Lustre: server umount lustre-MDT0000 complete [11738.652453] LustreError: 504598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146846 with bad export cookie 3829087646315872779 [11738.655114] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11738.656306] LustreError: 504598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11738.782231] Lustre: server umount lustre-MDT0001 complete [11742.720565] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [11743.823305] Key type lgssc unregistered [11743.969512] LNet: 508522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11743.972269] LNetError: 508522:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11743.978398] LNet: Removed LNI 192.168.202.145@tcp [11744.322320] Key type .llcrypt unregistered [11744.323431] Key type ._llcrypt unregistered [11752.077782] Key type ._llcrypt registered [11752.079016] Key type .llcrypt registered [11752.114106] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [11752.513367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11752.519925] alg: No test for adler32 (adler32-zlib) [11753.362654] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [11753.445442] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [11755.023152] Key type lgssc registered [11755.346974] Lustre: Echo OBD driver; http://www.lustre.org/ [11757.614347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11757.619531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11758.750243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11759.986377] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11762.607066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11762.724906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11764.152341] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11766.567092] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11768.388305] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11770.662200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11770.782077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11772.781510] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11775.195426] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11789.614127] loop0: detected capacity change from 0 to 204800000 [11789.675632] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11789.681452] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11789.685643] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11791.807139] loop0: detected capacity change from 0 to 204800000 [11791.867610] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11791.872818] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11791.879996] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11793.719965] loop0: detected capacity change from 0 to 204800000 [11795.677501] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11795.735305] Lustre: fs15246-OST0000: new disk, initializing [11795.737276] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [11795.739732] Lustre: 513384:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11795.758717] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11797.801360] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11801.663923] loop1: detected capacity change from 0 to 204800000 [11803.534259] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11803.578229] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [11803.604701] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [11803.650387] Lustre: fs15246-MDT0000: new disk, initializing [11803.704192] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11803.717156] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11803.757448] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11804.768715] Lustre: 514506:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000b3a57e11 x1869765002952576/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783146928 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [11805.105680] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11813.855435] LustreError: 514506:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [11813.859367] LustreError: 514506:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [11813.862154] LustreError: 514506:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [11816.935703] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11816.940372] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [11818.583394] Lustre: server umount fs15246-MDT0000 complete [11821.723614] Lustre: server umount fs15246-OST0000 complete [11825.119850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11825.123392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11825.130803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11827.168811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11827.169232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11831.251491] Lustre: server umount lustre-OST0000 complete [11834.336430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11834.341719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11834.350417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11834.355629] Lustre: Skipped 1 previous similar message [11838.682362] Lustre: server umount lustre-MDT0000 complete [11839.935467] LustreError: 509755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146947 with bad export cookie 9526069509693513794 [11839.939958] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11839.941058] LustreError: 509755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11840.074585] Lustre: server umount lustre-MDT0001 complete [11843.818783] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [11844.944519] Key type lgssc unregistered [11845.084335] LNet: 517599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11845.087460] LNetError: 517599:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11845.094414] LNet: Removed LNI 192.168.202.145@tcp [11845.430606] Key type .llcrypt unregistered [11845.431847] Key type ._llcrypt unregistered [11853.265076] Key type ._llcrypt registered [11853.266121] Key type .llcrypt registered [11853.304329] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [11853.665338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11853.720680] alg: No test for adler32 (adler32-zlib) [11854.568841] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [11854.655404] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [11856.231145] Key type lgssc registered [11856.584863] Lustre: Echo OBD driver; http://www.lustre.org/ [11868.803425] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [11872.154136] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [11883.605558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11883.612149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11884.691890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11884.701839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11884.745624] Lustre: lustre-MDT0000: new disk, initializing [11884.768986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11884.775439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11886.103028] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11890.150705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11890.190940] Lustre: 522035:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11890.213549] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11890.215052] Lustre: Skipped 1 previous similar message [11890.250117] Lustre: lustre-MDT0001: new disk, initializing [11890.282615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11890.294947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11890.297953] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11891.588619] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11895.275820] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11897.093600] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11899.314412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11899.415481] Lustre: lustre-OST0000: new disk, initializing [11899.417040] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11899.420312] Lustre: 524000:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11899.463151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11901.352582] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11904.493128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11904.498556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11904.512808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11905.005479] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11906.463553] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11906.549684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11907.927479] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11907.998579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11909.601637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11909.602215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11909.603645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11909.603717] LustreError: Skipped 1 previous similar message [11909.606817] Lustre: Skipped 1 previous similar message [11910.671735] Lustre: server umount lustre-OST0000 complete [11914.720904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11914.721616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11914.726096] Lustre: Skipped 1 previous similar message [11914.729179] Lustre: Skipped 2 previous similar messages [11918.035984] Lustre: server umount lustre-MDT0000 complete [11919.269774] LustreError: 522025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147026 with bad export cookie 17990270681108255709 [11919.274154] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11919.276430] LustreError: 522025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11919.395656] Lustre: server umount lustre-MDT0001 complete [11927.457559] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [11930.956522] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [11941.540790] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11685 sec ========= 02:37:28 (1783147048) [11942.155962] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:37:28 (1783147048) === [11943.365525] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:37:29 (1783147049) === [11956.906174] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [11957.938387] Key type lgssc unregistered [11958.067321] LNet: 530629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11958.070192] LNetError: 530629:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11958.079376] LNet: Removed LNI 192.168.202.145@tcp [11958.426748] Key type .llcrypt unregistered [11958.427689] Key type ._llcrypt unregistered