[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 498261121 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2608MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002020] Switched APIC routing to physical x2apic. [ 0.003014] kvm-guest: setup PV IPIs [ 0.005590] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009145] LSM: Security Framework initializing [ 0.010045] Yama: becoming mindful. [ 0.010897] SELinux: Initializing. [ 0.011000] *** VALIDATE selinux *** [ 0.018152] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022237] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023165] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025097] *** VALIDATE tmpfs *** [ 0.027354] *** VALIDATE proc *** [ 0.028184] *** VALIDATE cgroup *** [ 0.029007] *** VALIDATE cgroup2 *** [ 0.030219] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031122] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033032] Spectre V2 : User space: Vulnerable [ 0.034015] Speculative Store Bypass: Vulnerable [ 0.037178] debug: unmapping init [mem 0xffffffffa7459000-0xffffffffa7460fff] [ 0.039909] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040594] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041021] ... version: 2 [ 0.041969] ... bit width: 48 [ 0.042011] ... generic registers: 4 [ 0.042940] ... value mask: 0000ffffffffffff [ 0.043014] ... max period: 00007fffffffffff [ 0.044015] ... fixed-purpose events: 3 [ 0.044981] ... event mask: 000000070000000f [ 0.045327] rcu: Hierarchical SRCU implementation. [ 0.047205] smp: Bringing up secondary CPUs ... [ 0.048504] x86: Booting SMP configuration: [ 0.049023] .... node #0, CPUs: #1 #2 #3 [ 0.052077] smp: Brought up 1 node, 4 CPUs [ 0.054009] smpboot: Max logical packages: 1 [ 0.055017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.134403] node 0 deferred pages initialised in 77ms [ 0.137294] devtmpfs: initialized [ 0.138290] x86/mm: Memory block size: 128MB [ 0.141874] gcov: version magic: 0x41383552 [ 0.145217] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150397] pinctrl core: initialized pinctrl subsystem [ 0.152210] [ 0.152732] ************************************************************* [ 0.155014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157014] ** ** [ 0.160016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162012] ** ** [ 0.164015] ** This means that this kernel is built to expose internal ** [ 0.167017] ** IOMMU data structures, which may compromise security on ** [ 0.169014] ** your system. ** [ 0.171016] ** ** [ 0.174015] ** If you see this message and you are not debugging the ** [ 0.176015] ** kernel, report this immediately to your vendor! ** [ 0.179014] ** ** [ 0.181015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182014] ************************************************************* [ 0.185790] NET: Registered protocol family 16 [ 0.187428] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196127] cpuidle: using governor menu [ 0.197821] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202226] PCI: Using configuration type 1 for base access [ 0.204151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212151] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.213047] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.215131] cryptd: max_cpu_qlen set to 1000 [ 0.218207] ACPI: Added _OSI(Module Device) [ 0.220030] ACPI: Added _OSI(Processor Device) [ 0.222026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.223012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228147] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235135] ACPI: Interpreter enabled [ 0.236066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238015] ACPI: Using IOAPIC for interrupt routing [ 0.240123] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.244000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253720] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.256036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.257019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.260086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.264516] acpiphp: Slot [2] registered [ 0.266167] acpiphp: Slot [5] registered [ 0.267109] acpiphp: Slot [6] registered [ 0.268132] acpiphp: Slot [7] registered [ 0.269130] acpiphp: Slot [8] registered [ 0.271137] acpiphp: Slot [9] registered [ 0.273101] acpiphp: Slot [10] registered [ 0.274166] acpiphp: Slot [3] registered [ 0.276180] acpiphp: Slot [4] registered [ 0.277117] acpiphp: Slot [11] registered [ 0.279142] acpiphp: Slot [12] registered [ 0.281115] acpiphp: Slot [13] registered [ 0.282114] acpiphp: Slot [14] registered [ 0.283136] acpiphp: Slot [15] registered [ 0.285164] acpiphp: Slot [16] registered [ 0.286110] acpiphp: Slot [17] registered [ 0.288123] acpiphp: Slot [18] registered [ 0.289133] acpiphp: Slot [19] registered [ 0.291116] acpiphp: Slot [20] registered [ 0.292123] acpiphp: Slot [21] registered [ 0.293111] acpiphp: Slot [22] registered [ 0.295138] acpiphp: Slot [23] registered [ 0.296116] acpiphp: Slot [24] registered [ 0.298152] acpiphp: Slot [25] registered [ 0.299105] acpiphp: Slot [26] registered [ 0.300121] acpiphp: Slot [27] registered [ 0.302128] acpiphp: Slot [28] registered [ 0.304104] acpiphp: Slot [29] registered [ 0.305114] acpiphp: Slot [30] registered [ 0.307105] acpiphp: Slot [31] registered [ 0.308065] PCI host bridge to bus 0000:00 [ 0.310039] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.312028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.315028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.316026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.318019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.321026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.322152] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325092] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.328223] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.339786] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.344989] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.348020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.350028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.352021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355472] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.358941] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.362047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.365820] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.371015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.384019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.390009] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.396118] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.402018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.409019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.423023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.435031] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.442016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.447016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.461018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.472050] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.478017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.483019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.499022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.506371] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.514078] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.520017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.537025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.545453] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.556027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.572022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.587018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.599740] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.606019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.613023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.632018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.641256] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.645409] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.647433] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.649399] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.651192] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.658035] iommu: Default domain type: Passthrough [ 0.660620] SCSI subsystem initialized [ 0.662138] ACPI: bus type USB registered [ 0.663245] usbcore: registered new interface driver usbfs [ 0.666085] usbcore: registered new interface driver hub [ 0.668090] usbcore: registered new device driver usb [ 0.670195] pps_core: LinuxPPS API ver. 1 registered [ 0.672015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.675064] PTP clock support registered [ 0.678063] EDAC MC: Ver: 3.0.0 [ 0.680089] PCI: Using ACPI for IRQ routing [ 0.681674] NetLabel: Initializing [ 0.683011] NetLabel: domain hash size = 128 [ 0.685013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.687088] NetLabel: unlabeled traffic allowed by default [ 0.689150] vgaarb: loaded [ 0.691296] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.693016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.700000] clocksource: Switched to clocksource kvm-clock [ 0.801156] VFS: Disk quotas dquot_6.6.0 [ 0.802790] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.805532] *** VALIDATE ramfs *** [ 0.806827] *** VALIDATE hugetlbfs *** [ 0.808141] pnp: PnP ACPI init [ 0.810023] pnp: PnP ACPI: found 6 devices [ 0.826033] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.829365] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.831388] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.833621] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.835982] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.838318] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.841310] NET: Registered protocol family 2 [ 0.843605] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.847650] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.850912] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.856173] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.859828] TCP: Hash tables configured (established 65536 bind 65536) [ 0.863220] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.866745] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.869955] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.873108] NET: Registered protocol family 1 [ 0.875959] RPC: Registered named UNIX socket transport module. [ 0.878569] RPC: Registered udp transport module. [ 0.880413] RPC: Registered tcp transport module. [ 0.882258] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.884884] NET: Registered protocol family 44 [ 0.886681] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.888944] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.891367] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.893948] PCI: CLS 0 bytes, default 64 [ 0.895772] Unpacking initramfs... [ 2.222689] debug: unmapping init [mem 0xffff9f3d7cc54000-0xffff9f3d7ffbffff] [ 2.225413] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.227024] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.229046] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.708725] Initialise system trusted keyrings [ 2.711060] Key type blacklist registered [ 2.713900] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.724440] zbud: loaded [ 2.727344] *** VALIDATE nfs *** [ 2.728746] *** VALIDATE nfs4 *** [ 2.730589] pstore: using deflate compression [ 2.734147] Platform Keyring initialized [ 2.844969] NET: Registered protocol family 38 [ 2.847259] Key type asymmetric registered [ 2.849124] Asymmetric key parser 'x509' registered [ 2.851300] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.854895] io scheduler mq-deadline registered [ 2.856620] io scheduler kyber registered [ 2.859763] io scheduler bfq registered [ 2.861986] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.865411] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.868758] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.871969] ACPI: Power Button [PWRF] [ 2.877630] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.884855] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.897226] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.907426] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.927096] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.956513] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.984366] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.989932] Non-volatile memory driver v1.3 [ 2.991837] Linux agpgart interface v0.103 [ 3.019461] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.022014] vda: detected capacity change from 0 to 75083776 [ 3.035910] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.038446] vdb: detected capacity change from 0 to 1073741824 [ 3.054232] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.056876] vdc: detected capacity change from 0 to 2621440000 [ 3.069683] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.071581] vdd: detected capacity change from 0 to 2621440000 [ 3.083791] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.085592] vde: detected capacity change from 0 to 4294967296 [ 3.098747] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.100935] vdf: detected capacity change from 0 to 4294967296 [ 3.105623] libphy: Fixed MDIO Bus: probed [ 3.111208] usbcore: registered new interface driver usbserial_generic [ 3.113700] usbserial: USB Serial support registered for generic [ 3.116100] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.120345] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.122244] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.124782] mousedev: PS/2 mouse device common for all mice [ 3.127485] rtc_cmos 00:05: RTC can wake from S4 [ 3.131985] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.132431] rtc_cmos 00:05: registered as rtc0 [ 3.136994] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.139903] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.140565] intel_pstate: CPU model not supported [ 3.146138] hid: raw HID events driver (C) Jiri Kosina [ 3.148397] usbcore: registered new interface driver usbhid [ 3.149203] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.149618] usbhid: USB HID core driver [ 3.153701] drop_monitor: Initializing network drop monitor service [ 3.155260] Initializing XFRM netlink socket [ 3.156450] NET: Registered protocol family 10 [ 3.159331] Segment Routing with IPv6 [ 3.160947] NET: Registered protocol family 17 [ 3.163105] mpls_gso: MPLS GSO support [ 3.168156] RAS: Correctable Errors collector initialized. [ 3.170415] AVX version of gcm_enc/dec engaged. [ 3.172117] AES CTR mode by8 optimization enabled [ 3.234781] sched_clock: Marking stable (3234763518, 0)->(4066162056, -831398538) [ 3.237296] registered taskstats version 1 [ 3.238938] Loading compiled-in X.509 certificates [ 3.240116] zswap: loaded using pool lzo/zbud [ 3.261463] Key type big_key registered [ 3.274111] Key type encrypted registered [ 3.275819] ima: No TPM chip found, activating TPM-bypass! [ 3.277870] ima: Allocated hash algorithm: sha1 [ 3.279622] ima: No architecture policies found [ 3.281340] evm: Initialising EVM extended attributes: [ 3.283278] evm: security.selinux [ 3.284436] evm: security.ima [ 3.285156] evm: security.capability [ 3.286022] evm: HMAC attrs: 0x1 [ 3.288545] rtc_cmos 00:05: setting system clock to 2026-09-03 15:07:07 UTC (1788448027) [ 3.294438] debug: unmapping init [mem 0xffffffffa8403000-0xffffffffa85fffff] [ 3.297729] debug: unmapping init [mem 0xffffffffa7182000-0xffffffffa7458fff] [ 3.305120] Write protecting the kernel read-only data: 28672k [ 3.308449] debug: unmapping init [mem 0xffffffffa5803000-0xffffffffa59fffff] [ 3.311365] debug: unmapping init [mem 0xffffffffa6114000-0xffffffffa61fffff] [ 3.345337] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.350621] systemd[1]: Detected virtualization kvm. [ 3.351762] systemd[1]: Detected architecture x86-64. [ 3.352770] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.377578] systemd[1]: No hostname configured. [ 3.378529] systemd[1]: Set hostname to . [ 3.379677] random: systemd: uninitialized urandom read (16 bytes read) [ 3.381148] systemd[1]: Initializing machine ID from random generator. [ 3.406726] random: ln: uninitialized urandom read (6 bytes read) [ 3.500534] random: systemd: uninitialized urandom read (16 bytes read) [ 3.503854] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.508435] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.513270] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.034203] device-mapper: uevent: version 1.0.3 [ 4.036808] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 4.534325] random: fast init done [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.675544] virtio_net virtio0 ens2: renamed from eth0 [ 4.705093] scsi host0: ata_piix [ 4.727367] scsi host1: ata_piix [ 4.728864] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.731176] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.013497] dracut-initqueue[590]: RTNETLINK answers: File exists [ 9.791529] random: crng init done [ 9.792885] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.031804] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.111115] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.372891] SELinux: Disabled at runtime. [ 11.434568] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.440561] systemd[1]: Detected virtualization kvm. [ 11.441860] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.897511] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.900208] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.903927] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.906660] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.909088] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.916891] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.921231] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ 12.012369] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.269360] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.626212] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.645802] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.749915] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.772439] EDAC sbridge: Ver: 1.1.2 [ 14.160809] Key type dns_resolver registered [ 14.473610] NFS: Registering the id_resolver key type [ 14.474780] Key type id_resolver registered [ 14.475568] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg241-server login: [ 42.625353] hrtimer: interrupt took 2754534 ns [ 57.444377] libcfs: loading out-of-tree module taints kernel. [ 57.556476] Key type ._llcrypt registered [ 57.559796] Key type .llcrypt registered [ 57.805974] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 74.449620] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 75.851284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.869313] alg: No test for adler32 (adler32-zlib) [ 77.001405] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 78.094173] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 79.847267] Key type lgssc registered [ 81.157759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 103.889934] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 149.543908] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 165.399740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 165.436716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 166.777660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 166.803306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 166.896668] Lustre: lustre-MDT0000: new disk, initializing [ 167.015689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.051707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 172.463684] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.940297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.099391] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 186.127523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 186.134848] Lustre: Skipped 1 previous similar message [ 186.277068] Lustre: lustre-MDT0001: new disk, initializing [ 186.416255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 186.452171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 186.469157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 191.918834] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.493189] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 207.259962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.665457] Lustre: lustre-OST0000: new disk, initializing [ 207.673356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 207.688528] Lustre: 8458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.812496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.860855] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.666729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 217.675978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 217.802741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 226.922441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 227.067576] Lustre: lustre-OST0001: new disk, initializing [ 227.069369] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 227.072785] Lustre: 9531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 227.126755] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 233.547872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 233.558534] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 233.663262] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 234.997455] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.798206] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 259.791151] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 265.756983] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing check_logdir /tmp/testlogs/ [ 271.227227] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing yml_node [ 277.009825] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 280.007314] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 282.711829] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 284.537346] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 11:11:47 EDT 2026 [ 304.936661] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 306.885074] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 324.579559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 324.589892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.606524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.602329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.612624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.614841] Lustre: Skipped 1 previous similar message [ 325.640217] Lustre: Skipped 1 previous similar message [ 330.721284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.731298] Lustre: Skipped 5 previous similar messages [ 335.841472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.852766] Lustre: Skipped 3 previous similar messages [ 338.399156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 339.079384] Lustre: server umount lustre-MDT0000 complete [ 343.357799] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448367 with bad export cookie 7215586816546441662 [ 343.363200] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 343.372618] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 346.080542] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.083473] LustreError: 7942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.084164] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.110720] Lustre: Skipped 2 previous similar messages [ 346.174318] LustreError: 7942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 348.130178] LustreError: 6528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.151463] LustreError: 6528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 349.796528] Lustre: server umount lustre-MDT0001 complete [ 354.071895] Lustre: server umount lustre-OST0000 complete [ 358.494432] Lustre: server umount lustre-OST0001 complete [ 366.407228] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 375.870902] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 413.662647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.930631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 413.974762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 414.181919] Lustre: lustre-MDT0000: new disk, initializing [ 414.328293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.346314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 420.152580] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 434.340392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.398876] Lustre: 15088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 434.434148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 434.438414] Lustre: Skipped 1 previous similar message [ 434.490222] Lustre: lustre-MDT0001: new disk, initializing [ 434.582441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 434.638733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 434.657389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 439.355174] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 452.372537] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 458.955317] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 466.004602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 466.541150] Lustre: lustre-OST0000: new disk, initializing [ 466.546984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 466.554770] Lustre: 17059:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 466.679687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.021135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 468.041964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 468.163569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 472.528837] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 481.145335] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 486.633310] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 486.878106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 492.493390] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 492.830693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 499.168852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 499.180991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.198800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 499.200951] Lustre: Skipped 1 previous similar message [ 499.223739] Lustre: Skipped 4 previous similar messages [ 501.150658] Lustre: server umount lustre-OST0000 complete [ 506.336634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 506.344284] LustreError: Skipped 1 previous similar message [ 506.348188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.866557] Lustre: server umount lustre-MDT0000 complete [ 514.543946] LustreError: 15096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.559331] LustreError: 15096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 515.665343] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448539 with bad export cookie 7215586816546443195 [ 515.679237] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 516.002443] Lustre: server umount lustre-MDT0001 complete [ 520.201982] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 11:15:42 (1788448542) [ 522.849958] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 524.895649] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 11:15:47 (1788448547) [ 533.309064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 533.809212] LustreError: 19468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.916714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 539.108000] LustreError: 19469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.283203] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 549.213793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 549.345294] LustreError: 19468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.372659] LustreError: 19468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 554.937458] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 563.380686] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 569.252788] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 576.747347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 584.461953] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 592.160760] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 605.152913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 605.159080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.164366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 605.164375] Lustre: Skipped 3 previous similar messages [ 605.180669] Lustre: Skipped 2 previous similar messages [ 608.754931] Lustre: server umount lustre-OST0000 complete [ 619.016164] Lustre: server umount lustre-MDT0000 complete [ 620.512696] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 620.524707] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 623.341374] LustreError: 19450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448647 with bad export cookie 7215586816546444077 [ 623.356379] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 623.360064] LustreError: 19450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 623.722649] Lustre: server umount lustre-MDT0001 complete [ 639.189844] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 642.773928] Key type lgssc unregistered [ 643.123644] LNet: 23003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.133175] LNetError: 23003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 643.158929] LNet: Removed LNI 192.168.202.141@tcp [ 644.120172] Key type .llcrypt unregistered [ 644.124056] Key type ._llcrypt unregistered [ 668.526107] Key type ._llcrypt registered [ 668.527748] Key type .llcrypt registered [ 668.637279] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 670.122719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 670.169982] alg: No test for adler32 (adler32-zlib) [ 671.203710] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 671.510843] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 673.191786] Key type lgssc registered [ 674.474317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 683.216566] loop: module loaded [ 683.330124] loop0: detected capacity change from 0 to 204800000 [ 683.578483] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 683.587878] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 683.887855] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 690.428823] loop0: detected capacity change from 0 to 204800000 [ 690.667715] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 690.678953] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 690.697537] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 696.103885] loop0: detected capacity change from 0 to 204800000 [ 701.755463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 701.778798] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 703.107665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 703.165478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 703.222856] Lustre: lustre-MDT0000: new disk, initializing [ 703.271265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 703.285677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 707.780636] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 716.899524] loop1: detected capacity change from 0 to 204800000 [ 722.928722] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 723.263070] Lustre: lustre-OST07c6: new disk, initializing [ 723.266832] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 723.271305] Lustre: Skipped 1 previous similar message [ 723.276703] Lustre: 26191:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 723.346250] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 729.742271] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 731.754353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 731.762184] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 731.815688] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 735.699656] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 735.712100] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 735.723437] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 735.733310] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 735.748200] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 735.755310] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 754.209696] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 754.218250] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 754.223655] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 754.228295] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 754.241971] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 754.248688] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 758.858124] Lustre: 25177:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 758.874746] Lustre: 25177:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 758.882522] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 758.895083] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.904611] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 758.920756] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.930051] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 758.938059] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.949770] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 758.962377] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.981322] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 758.987868] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.205420] Lustre: 26860:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 762.214719] Lustre: 26860:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 762.224681] Lustre: 26860:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 762.234772] Lustre: 26860:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.246245] Lustre: 26860:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 762.249467] Lustre: 26860:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.264758] Lustre: 26860:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 762.274897] Lustre: 26860:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.282296] Lustre: 26860:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 762.293767] Lustre: 26860:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.303258] Lustre: 26860:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 762.309758] Lustre: 26860:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.902295] Lustre: 25177:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 766.909233] Lustre: 25177:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 766.913178] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 766.919761] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.925117] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 766.930856] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.936585] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 766.940557] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.945046] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 766.952196] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.956372] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 766.959635] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.924558] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 777.930073] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 777.934260] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 777.938873] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 777.944495] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 777.949683] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 777.956060] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 777.961648] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 777.966304] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 777.971321] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 777.976133] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.981126] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 823.781647] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 823.802375] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 828.899157] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 829.923838] Lustre: server umount lustre-OST07c6 complete [ 839.597913] Lustre: server umount lustre-MDT0000 complete [ 857.662775] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 860.821827] Key type lgssc unregistered [ 861.245469] LNet: 28751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.264974] LNetError: 28751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.296742] LNet: Removed LNI 192.168.202.141@tcp [ 862.362425] Key type .llcrypt unregistered [ 862.366105] Key type ._llcrypt unregistered [ 882.228639] Key type ._llcrypt registered [ 882.230884] Key type .llcrypt registered [ 882.353440] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 883.790650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 883.814343] alg: No test for adler32 (adler32-zlib) [ 884.891694] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 885.165050] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 886.881155] Key type lgssc registered [ 887.991028] Lustre: Echo OBD driver; http://www.lustre.org/ [ 895.947972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 895.957981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 897.321124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 901.962180] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 907.759933] LustreError: 30017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 907.776535] LustreError: 30017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 911.388890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.838202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 917.248473] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 924.481995] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 930.473737] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 937.707645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 938.109642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 942.194502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 942.985914] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 949.591873] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 972.268500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 972.276315] Lustre: Skipped 1 previous similar message [ 972.277240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 972.285075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 977.387673] LustreError: 31841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 977.410430] LustreError: 31841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 977.562494] Lustre: server umount lustre-OST0000 complete [ 982.497713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 982.500682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 982.508561] Lustre: Skipped 1 previous similar message [ 982.528150] Lustre: Skipped 2 previous similar messages [ 987.532833] Lustre: server umount lustre-MDT0000 complete [ 987.620981] LustreError: 30018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 987.642462] LustreError: 30018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 991.559110] LustreError: 29998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449015 with bad export cookie 16705184834096680532 [ 991.565655] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 991.576460] LustreError: 29998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 991.879486] Lustre: server umount lustre-MDT0001 complete [ 1001.811464] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1004.145181] Key type lgssc unregistered [ 1004.478754] LNet: 33772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.488689] LNetError: 33772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.505792] LNet: Removed LNI 192.168.202.141@tcp [ 1005.343676] Key type .llcrypt unregistered [ 1005.345858] Key type ._llcrypt unregistered [ 1024.235597] Key type ._llcrypt registered [ 1024.248885] Key type .llcrypt registered [ 1024.391775] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1025.405259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1025.457060] alg: No test for adler32 (adler32-zlib) [ 1026.613874] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 1026.847941] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1028.527276] Key type lgssc registered [ 1029.794041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1037.004192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1037.033617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1038.454609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1042.918902] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1048.035280] LustreError: 35039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.071413] LustreError: 35039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1050.687564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.087618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1054.943467] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1061.768656] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1067.898974] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1074.968559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1075.396399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.738882] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1083.651334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1087.399503] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1102.329053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1102.339190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1102.349339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1104.435617] Lustre: server umount lustre-OST0000 complete [ 1109.480453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1109.492678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1109.509171] Lustre: Skipped 1 previous similar message [ 1109.520742] Lustre: Skipped 1 previous similar message [ 1114.595054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.604181] Lustre: Skipped 1 previous similar message [ 1118.872720] Lustre: server umount lustre-MDT0000 complete [ 1119.715292] LustreError: 35040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1119.743156] LustreError: 35040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1122.991256] LustreError: 35021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449147 with bad export cookie 17735323020755310817 [ 1123.002665] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1123.009746] LustreError: 35021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1123.427346] Lustre: server umount lustre-MDT0001 complete [ 1136.741704] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1140.126807] Key type lgssc unregistered [ 1140.563801] LNet: 38593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.569298] LNetError: 38593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1141.608748] LNet: Removed LNI 192.168.202.141@tcp [ 1142.566726] Key type .llcrypt unregistered [ 1142.568314] Key type ._llcrypt unregistered [ 1167.202749] Key type ._llcrypt registered [ 1167.205547] Key type .llcrypt registered [ 1167.387712] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1169.140960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1169.195691] alg: No test for adler32 (adler32-zlib) [ 1170.327266] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 1170.695783] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1172.479299] Key type lgssc registered [ 1173.765220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1181.726501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1181.746959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1183.233879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1188.160112] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1193.446906] LustreError: 40009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.460426] LustreError: 40009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1196.375817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.702367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1200.752530] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1208.540342] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1215.407894] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1224.436031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1225.074448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.189880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1231.442328] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1238.223138] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1246.708458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1246.718132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1246.724508] Lustre: Skipped 1 previous similar message [ 1246.734217] Lustre: Skipped 2 previous similar messages [ 1249.813528] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 1251.811054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1251.821683] Lustre: Skipped 2 previous similar messages [ 1252.873132] Lustre: server umount lustre-MDT0000 complete [ 1254.912807] LustreError: 42587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.487970] LustreError: 39990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449280 with bad export cookie 2752639245601506266 [ 1256.503713] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1256.509524] LustreError: 39990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1256.928934] LustreError: 40004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.930625] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1256.985211] Lustre: Skipped 1 previous similar message [ 1256.994961] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1260.020693] LustreError: 42589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.048830] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1262.062547] Lustre: Skipped 1 previous similar message [ 1262.927345] Lustre: server umount lustre-MDT0001 complete [ 1279.723516] Lustre: server umount lustre-OST0000 complete [ 1297.689875] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1300.835686] Key type lgssc unregistered [ 1301.101608] LNet: 43868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.115370] LNetError: 43868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1301.134614] LNet: Removed LNI 192.168.202.141@tcp [ 1301.890682] Key type .llcrypt unregistered [ 1301.891976] Key type ._llcrypt unregistered [ 1327.044693] Key type ._llcrypt registered [ 1327.046954] Key type .llcrypt registered [ 1327.281401] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1328.897269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1328.989800] alg: No test for adler32 (adler32-zlib) [ 1330.118701] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 1330.424724] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1332.183561] Key type lgssc registered [ 1333.562703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1342.090687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1342.107912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1343.635860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1348.681269] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1353.194529] LustreError: 45291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.226840] LustreError: 45291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1357.905744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.204525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1362.611178] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1370.523710] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1376.454399] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1384.415838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1384.943472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.045038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1393.195291] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1402.381331] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1410.570359] Lustre: lustre-OST0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 1413.609907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.613311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1413.626630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1413.638260] Lustre: Skipped 1 previous similar message [ 1415.676126] Lustre: lustre-OST0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 1415.685542] Lustre: Skipped 1 previous similar message [ 1418.721516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1418.734052] Lustre: Skipped 1 previous similar message [ 1423.329249] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1423.596304] Lustre: server umount lustre-OST0000 complete [ 1444.320803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.337255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1444.339853] Lustre: Skipped 1 previous similar message [ 1448.113243] Lustre: server umount lustre-MDT0000 complete [ 1450.475574] LustreError: 45286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1450.504207] LustreError: 45286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1452.485107] LustreError: 45272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449476 with bad export cookie 12485193116701988041 [ 1452.490636] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1452.501109] LustreError: 45272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1459.172881] Lustre: server umount lustre-MDT0001 complete [ 1471.924901] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1474.969273] Key type lgssc unregistered [ 1475.339806] LNet: 49003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1475.344400] LNetError: 49003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.356288] LNet: Removed LNI 192.168.202.141@tcp [ 1476.496529] Key type .llcrypt unregistered [ 1476.499786] Key type ._llcrypt unregistered [ 1503.468414] Key type ._llcrypt registered [ 1503.470592] Key type .llcrypt registered [ 1503.577404] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1505.172782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1505.186238] alg: No test for adler32 (adler32-zlib) [ 1506.221042] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 1506.491110] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1508.223160] Key type lgssc registered [ 1509.264239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1517.470315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1517.515042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1519.154025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1525.503208] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1529.319596] LustreError: 50418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1529.343197] LustreError: 50418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1534.432590] LustreError: 50417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1535.821556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1536.237112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1541.890937] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1548.258040] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1555.039263] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1563.383963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1563.946688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1565.002730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1571.984311] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1580.876045] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1587.876269] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1591.874481] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1604.071288] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0311h33m46s [ 1605.352536] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1608.160306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1608.170493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.185562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1609.281077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1609.297284] Lustre: Skipped 5 previous similar messages [ 1610.354981] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 1613.843479] Lustre: server umount lustre-MDT0000 complete [ 1615.338344] LustreError: 51145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1615.359444] LustreError: 51145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1617.596516] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1621.475893] LustreError: 50417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.505862] LustreError: 50417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1625.139281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1625.346293] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1625.689556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1625.753547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1626.735576] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1630.236298] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1633.558594] Lustre: DEBUG MARKER: Wait for df (36701) ... [ 1635.148727] Lustre: DEBUG MARKER: done [ 1640.937731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1640.945447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1640.973079] Lustre: Skipped 2 previous similar messages [ 1640.984855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1642.498124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.179300] Lustre: server umount lustre-OST0000 complete [ 1652.718964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1652.725785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.733481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1652.733494] Lustre: Skipped 2 previous similar messages [ 1652.767866] Lustre: Skipped 1 previous similar message [ 1655.402468] Lustre: server umount lustre-MDT0000 complete [ 1659.826689] LustreError: 50399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449684 with bad export cookie 1258553259140833273 [ 1659.845921] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1660.080323] Lustre: server umount lustre-MDT0001 complete [ 1673.141240] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1676.456571] Key type lgssc unregistered [ 1676.896834] LNet: 55349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.904589] LNetError: 55349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.919518] LNet: Removed LNI 192.168.202.141@tcp [ 1677.874340] Key type .llcrypt unregistered [ 1677.882769] Key type ._llcrypt unregistered [ 1723.037687] Key type ._llcrypt registered [ 1723.039374] Key type .llcrypt registered [ 1723.145073] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1724.235898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1724.382390] alg: No test for adler32 (adler32-zlib) [ 1725.639348] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 1726.018394] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1727.727709] Key type lgssc registered [ 1729.377402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1737.479726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1737.496397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1738.834359] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1738.857027] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1739.036967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.986331] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1753.668343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1753.797966] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1753.845024] Lustre: 57555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1754.279520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1759.630447] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1768.380663] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1774.208838] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1780.315468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1780.434642] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1780.624315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1786.201651] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1787.515523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1792.716687] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1799.581332] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0311h37m02s [ 1801.096682] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1808.301069] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1811.034748] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1811.104889] LustreError: 57563:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1831.464082] Lustre: lustre-MDT0000: Client 02cf776b-51d0-4a37-b508-a6594bfc36fe (at 192.168.202.41@tcp) reconnecting [ 1851.183141] LustreError: 57563:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1854.759931] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1857.472419] Lustre: DEBUG MARKER: done [ 1864.161535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1864.174804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.182667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1869.842319] Lustre: server umount lustre-OST0000 complete [ 1874.914189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1874.922748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1874.940817] Lustre: Skipped 2 previous similar messages [ 1874.961208] Lustre: Skipped 2 previous similar messages [ 1876.004701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1876.020137] Lustre: Skipped 1 previous similar message [ 1880.848606] Lustre: server umount lustre-MDT0000 complete [ 1882.081074] LustreError: 57567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1882.116829] LustreError: 57567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1885.493695] LustreError: 57546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449909 with bad export cookie 4475476239159852499 [ 1885.500560] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1885.506898] LustreError: 57546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1885.934194] Lustre: server umount lustre-MDT0001 complete [ 1897.927097] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1901.326697] Key type lgssc unregistered [ 1901.706678] LNet: 61881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.718366] LNetError: 61881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.745847] LNet: Removed LNI 192.168.202.141@tcp [ 1902.568427] Key type .llcrypt unregistered [ 1902.570684] Key type ._llcrypt unregistered [ 1949.316355] Key type ._llcrypt registered [ 1949.320927] Key type .llcrypt registered [ 1949.449580] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1950.776234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1950.793653] alg: No test for adler32 (adler32-zlib) [ 1951.959811] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 1952.196986] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1953.903173] Key type lgssc registered [ 1955.174528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1958.863445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1958.909985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.255433] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1960.286760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1960.539270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.652633] Lustre: Failing over lustre-MDT0000 [ 1962.101353] Lustre: server umount lustre-MDT0000 complete [ 1968.066437] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:39:50 (1788449990) [ 1974.824638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1975.501150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.521479] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1990.293975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1990.429797] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1990.459836] Lustre: 64678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1990.854869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1995.762844] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2003.895219] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2009.528720] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2016.044624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2016.274703] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2016.455309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.008763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2022.854341] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2029.604901] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2035.125581] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2040.525293] Lustre: server umount lustre-MDT0000 complete [ 2042.337835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2042.343542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.367867] LustreError: 64689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2042.387336] LustreError: 64689:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2042.856332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.884082] LustreError: 64686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2042.910849] LustreError: 64686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2043.890616] LustreError: 64671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450068 with bad export cookie 1514082667033292223 [ 2043.904675] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2043.913122] LustreError: 64671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2044.309684] Lustre: server umount lustre-MDT0001 complete [ 2047.518510] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2051.634067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2060.348385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.351140] Lustre: 63597:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450072/real 1788450072] req@000000000f0fda38 x1875325722071296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788450088 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2064.373826] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.385902] Lustre: Skipped 1 previous similar message [ 2065.439206] Lustre: 63597:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450073/real 1788450073] req@00000000bad893f4 x1875325722071424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788450089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2069.471240] Lustre: 63599:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450077/real 1788450077] req@00000000d2557ac2 x1875325722071552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788450093 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2069.536000] LustreError: 63596:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000002d62410 x1875325722072448/t0(0) o250->MGC192.168.202.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2069.984704] LustreError: 66526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2070.088135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.205089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2074.037253] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2074.208648] LustreError: 66527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.236790] LustreError: 66527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2075.231797] Lustre: 63598:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450083/real 1788450083] req@00000000d484c193 x1875325722071936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788450099 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2075.270038] Lustre: 63598:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2079.714897] LustreError: 68448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.741391] LustreError: 68448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2083.030767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2083.463499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2084.532469] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2087.950364] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2094.264677] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2099.453213] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2105.315067] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.319747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.324228] Lustre: Skipped 1 previous similar message [ 2107.441950] Lustre: server umount lustre-MDT0000 complete [ 2110.438969] LustreError: 68995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2110.463458] LustreError: 68995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2111.173955] LustreError: 64669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450135 with bad export cookie 1514082667033294498 [ 2111.181126] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.186901] LustreError: 64669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2111.545587] Lustre: server umount lustre-MDT0001 complete [ 2117.925696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2127.509108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.914663] Lustre: 63599:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450139/real 1788450139] req@00000000c90f5b29 x1875325722097536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788450155 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2130.954566] Lustre: 63599:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2130.966447] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.980457] Lustre: Skipped 1 previous similar message [ 2136.098834] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x15031a29e9187c05 [ 2136.110574] Lustre: MGC192.168.202.141@tcp: Connection restored to 0@lo (at 0@lo) [ 2136.122898] Lustre: Skipped 1 previous similar message [ 2136.472571] LustreError: 69183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.614165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.663256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2141.102645] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2143.711671] Lustre: 63600:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450151/real 1788450151] req@0000000033408f02 x1875325722098176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788450167 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2143.743365] Lustre: 63600:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2149.847347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2155.070618] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2155.509153] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2160.681050] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2163.195184] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2169.949238] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2181.092384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2181.093946] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2181.111768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2181.111777] Lustre: Skipped 2 previous similar messages [ 2181.141995] Lustre: Skipped 2 previous similar messages [ 2184.737113] Lustre: server umount lustre-MDT0000 complete [ 2186.211274] LustreError: 66526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2186.223424] LustreError: 66526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2188.095376] LustreError: 64669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450212 with bad export cookie 1514082667033295877 [ 2188.100386] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2188.105555] LustreError: 64669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2188.325965] Lustre: server umount lustre-MDT0001 complete [ 2194.874115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2197.600098] Lustre: DEBUG MARKER: files compared the same [ 2209.579171] Lustre: server umount lustre-OST0000 complete [ 2223.841264] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 2225.885775] Key type lgssc unregistered [ 2226.082959] LNet: 74404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.088188] LNetError: 74404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2227.110870] LNet: Removed LNI 192.168.202.141@tcp [ 2227.754696] Key type .llcrypt unregistered [ 2227.758128] Key type ._llcrypt unregistered [ 2247.385950] Key type ._llcrypt registered [ 2247.386939] Key type .llcrypt registered [ 2247.436086] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2248.214201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2248.231787] alg: No test for adler32 (adler32-zlib) [ 2249.297583] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 2249.545938] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 2251.215371] Key type lgssc registered [ 2252.094257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2258.597750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2258.618952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.045211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.113611] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug +malloc all [ 2270.176404] LustreError: 75795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.186638] LustreError: 75795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2271.395596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2271.661824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2275.851943] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug +malloc all [ 2282.154628] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2287.758391] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2293.679886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2293.920751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2299.363951] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug +malloc all [ 2305.202494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2305.605572] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2315.743957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2315.747317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2315.815335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2315.828384] Lustre: Skipped 1 previous similar message [ 2320.864502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2320.871669] Lustre: Skipped 1 previous similar message [ 2321.458935] Lustre: server umount lustre-OST0000 complete [ 2328.032800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2328.048662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.072940] Lustre: Skipped 1 previous similar message [ 2328.099330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2331.109119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2331.110466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.501638] Lustre: server umount lustre-MDT0000 complete [ 2335.090509] LustreError: 75776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450359 with bad export cookie 8186212085016575536 [ 2335.097682] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2335.342120] Lustre: server umount lustre-MDT0001 complete [ 2344.560571] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 2347.132768] Key type lgssc unregistered [ 2347.442422] LNet: 79326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2347.448044] LNetError: 79326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2347.471442] LNet: Removed LNI 192.168.202.141@tcp [ 2348.251505] Key type .llcrypt unregistered [ 2348.256520] Key type ._llcrypt unregistered [ 2369.748981] Key type ._llcrypt registered [ 2369.751230] Key type .llcrypt registered [ 2369.872678] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2370.783844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2370.839942] alg: No test for adler32 (adler32-zlib) [ 2371.863767] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 2372.093312] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 2373.735189] Key type lgssc registered [ 2374.781785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2381.154678] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2381.186271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2396.767524] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2401.887341] LustreError: 80714:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.141@tcp: failed processing log, type 4: rc = -110 [ 2428.577895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2433.746472] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2442.006520] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2451.712932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2451.962783] Lustre: *** cfs_fail_loc=706, val=0*** [ 2452.263294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2452.356548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2457.163713] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2457.569691] LustreError: 80740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.583235] LustreError: 80740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2462.697684] LustreError: 80739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.718756] LustreError: 81800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.755331] LustreError: 81800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2465.214728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2465.863694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2470.132566] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2479.087518] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2486.039156] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2488.289953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2488.301776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.329150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2491.366133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2491.372953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2491.378172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.745843] Lustre: server umount lustre-OST0000 complete [ 2501.609201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2501.612862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2501.629650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2501.639761] Lustre: Skipped 2 previous similar messages [ 2503.910938] Lustre: server umount lustre-MDT0000 complete [ 2506.723124] LustreError: 81804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.757347] LustreError: 81804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2508.557094] LustreError: 83375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450532 with bad export cookie 12281584963260085979 [ 2508.565711] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2508.566575] LustreError: 83375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2509.238549] Lustre: server umount lustre-MDT0001 complete [ 2518.018164] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 2520.917902] Key type lgssc unregistered [ 2521.205314] LNet: 84328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2521.216849] LNetError: 84328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2522.287623] LNet: Removed LNI 192.168.202.141@tcp [ 2523.274810] Key type .llcrypt unregistered [ 2523.278262] Key type ._llcrypt unregistered [ 2550.548399] Key type ._llcrypt registered [ 2550.550805] Key type .llcrypt registered [ 2550.669449] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2552.239448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2552.307175] alg: No test for adler32 (adler32-zlib) [ 2553.393233] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 2553.679593] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 2555.399538] Key type lgssc registered [ 2556.439289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2564.530721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2564.542747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.694216] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2577.148255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2577.696595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2582.420809] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2583.009772] LustreError: 86428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2583.024138] LustreError: 86428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2587.109295] LustreError: 86431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2587.134379] LustreError: 86431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2589.890689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2590.147667] LustreError: 86428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.172075] LustreError: 86428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2590.180071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2595.300204] LustreError: 86427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.342767] LustreError: 86427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2596.810655] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2600.418962] LustreError: 86432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.443903] LustreError: 86432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2609.281280] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2609.306149] Lustre: MGS: Client 99a84d7a-1003-45db-9561-9c9dff2f17e6 (at 0@lo) reconnecting [ 2609.372645] LustreError: 86442:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2609.741676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.826300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2615.054099] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2630.113679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2630.116739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2630.119958] LustreError: Skipped 1 previous similar message [ 2630.121964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2630.141078] Lustre: Skipped 1 previous similar message [ 2633.192513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2633.200200] Lustre: Skipped 1 previous similar message [ 2635.347684] Lustre: server umount lustre-OST0000 complete [ 2640.352929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2640.375609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2640.400062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2640.408066] Lustre: Skipped 1 previous similar message [ 2643.434875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2643.435613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2643.462697] Lustre: Skipped 1 previous similar message [ 2645.624259] Lustre: server umount lustre-MDT0000 complete [ 2648.557862] LustreError: 87972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.597815] LustreError: 87972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2650.076488] Lustre: server umount lustre-MDT0001 complete [ 2654.369508] Lustre: server umount MGS complete [ 2666.406652] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:51:29 (1788450689) [ 2699.032587] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 2709.903067] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2749.618510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.917946] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2763.338162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2763.610756] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2763.647890] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2763.729452] Lustre: lustre-MDT0001: new disk, initializing [ 2763.812978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2763.834067] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2763.843339] Lustre: 93598:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2763.853455] LustreError: 93598:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2768.712265] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2778.965932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2779.296324] Lustre: lustre-OST0000: new disk, initializing [ 2779.300560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2779.308324] Lustre: 94365:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2779.403136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.402304] Lustre: 94658:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000056c84408 x1875326352271360/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788450826 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2786.690828] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2799.127202] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2806.178330] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2806.202228] Lustre: MGS: Client de4fe365-d259-4c71-a797-dc1e3f32f443 (at 0@lo) reconnecting [ 2806.266525] Lustre: 92867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2806.282970] LustreError: 93584:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2806.318712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2806.372628] Lustre: lustre-MDT0000: new disk, initializing [ 2806.491589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.525104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2806.547266] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2807.584469] Lustre: 95434:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000231bd4cd x1875326352278784/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788450847 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2810.925696] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2812.777230] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2816.685095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2816.690643] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2816.837355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2841.059991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.063593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2841.071064] Lustre: Skipped 1 previous similar message [ 2841.085560] Lustre: Skipped 1 previous similar message [ 2846.699338] Lustre: server umount lustre-OST0000 complete [ 2851.297167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2851.301846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.318573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2851.324735] Lustre: Skipped 1 previous similar message [ 2851.333741] Lustre: Skipped 3 previous similar messages [ 2856.753623] Lustre: server umount lustre-MDT0000 complete [ 2874.335446] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2874.505310] Lustre: server umount lustre-MDT0001 complete [ 2878.489271] Lustre: server umount MGS complete [ 2889.171422] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:55:11 (1788450911) [ 2907.857927] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 2910.752273] Key type lgssc unregistered [ 2911.014058] LNet: 97934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2911.026909] LNetError: 97934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2911.060910] LNet: Removed LNI 192.168.202.141@tcp [ 2911.827326] Key type .llcrypt unregistered [ 2911.830670] Key type ._llcrypt unregistered [ 2929.579368] Key type ._llcrypt registered [ 2929.582604] Key type .llcrypt registered [ 2929.649555] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2931.052858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2931.089366] alg: No test for adler32 (adler32-zlib) [ 2932.202764] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 2932.451316] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 2934.105068] Key type lgssc registered [ 2935.043898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2943.188832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2943.208317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.713657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2950.622945] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2954.217847] LustreError: 99152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2954.234822] LustreError: 99152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2959.329128] LustreError: 99151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2959.988979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2960.418083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2965.014383] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2973.517716] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2980.847372] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2991.789451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2992.655236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3001.065708] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3010.714532] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3014.219170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3024.353302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3024.358531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3024.381418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3026.355308] Lustre: server umount lustre-OST0000 complete [ 3032.032300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3032.040809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.050567] Lustre: Skipped 1 previous similar message [ 3032.055913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3032.058868] Lustre: Skipped 1 previous similar message [ 3034.603025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.603404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3038.002762] Lustre: server umount lustre-MDT0000 complete [ 3039.740968] LustreError: 99152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.777435] LustreError: 99152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3043.012085] LustreError: 99133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451067 with bad export cookie 8401854070348467462 [ 3043.017123] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3043.029273] LustreError: 99133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3043.673970] Lustre: server umount lustre-MDT0001 complete [ 3057.263522] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 3060.734665] Key type lgssc unregistered [ 3061.047764] LNet: 102700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3061.067930] LNetError: 102700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3061.088270] LNet: Removed LNI 192.168.202.141@tcp [ 3061.953718] Key type .llcrypt unregistered [ 3061.956098] Key type ._llcrypt unregistered [ 3079.224917] Key type ._llcrypt registered [ 3079.227242] Key type .llcrypt registered [ 3079.322708] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3080.331971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3080.366942] alg: No test for adler32 (adler32-zlib) [ 3081.607868] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 3081.873539] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 3083.609749] Key type lgssc registered [ 3084.689335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3090.953337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3090.958542] LustreError: 103777:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3090.992116] LustreError: 103780:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3090.996702] LustreError: 103777:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4969 [ 3090.999912] LustreError: 103777:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3091.011290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3091.011743] LustreError: 103777:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3092.559069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3100.224352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3100.478984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3105.007947] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3111.391405] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing lsmod [ 3114.547034] LustreError: 104978:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3114.634553] LustreError: 104988:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3114.650847] LustreError: 104978:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4908 [ 3114.660584] LustreError: 104978:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3114.668597] LustreError: 104978:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3114.676671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3115.167821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.244899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3121.122395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3121.128280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3121.153352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3122.662561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3122.667731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3122.686985] Lustre: Skipped 1 previous similar message [ 3124.865621] Lustre: server umount lustre-MDT0000 complete [ 3125.794734] LustreError: 103814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3125.841825] LustreError: 103814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3129.188972] LustreError: 103795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451153 with bad export cookie 4562155562404720324 [ 3129.199773] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3129.211780] LustreError: 103795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3129.952489] LustreError: 104250:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ceb9f43d x1875326906946816/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3130.290876] Lustre: server umount lustre-MDT0001 complete [ 3143.144659] Lustre: server umount lustre-OST0000 complete [ 3150.758320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3151.125684] LustreError: 106052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3151.207494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3155.043593] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3156.464253] LustreError: 106053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3161.582592] LustreError: 106052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3165.268425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3165.851110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3171.835258] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3180.146254] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3186.101724] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3192.524082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3192.805041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.891470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3198.706550] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3204.995811] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3216.353242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3216.360197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3216.370504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3216.373912] Lustre: Skipped 2 previous similar messages [ 3217.793166] Lustre: server umount lustre-OST0000 complete [ 3221.985313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3221.993398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3222.009098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3227.981061] Lustre: server umount lustre-MDT0000 complete [ 3230.691326] LustreError: 106996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.716671] LustreError: 106996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3232.195292] LustreError: 106034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451256 with bad export cookie 4562155562404721255 [ 3232.220870] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3242.677667] Lustre: server umount lustre-MDT0001 complete [ 3255.749748] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 3259.419788] Key type lgssc unregistered [ 3259.825732] LNet: 109598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3259.838479] LNetError: 109598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3259.861852] LNet: Removed LNI 192.168.202.141@tcp [ 3260.923253] Key type .llcrypt unregistered [ 3260.925895] Key type ._llcrypt unregistered [ 3287.426283] Key type ._llcrypt registered [ 3287.428581] Key type .llcrypt registered [ 3287.544328] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3289.310603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3289.399394] alg: No test for adler32 (adler32-zlib) [ 3290.578344] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 3290.925922] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 3292.712552] Key type lgssc registered [ 3294.180362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3303.003487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3303.040838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3304.621364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.413676] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3314.154377] LustreError: 111013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.178694] LustreError: 111013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3319.277150] LustreError: 111014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.457701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.834299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3325.102284] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3333.082439] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3338.122809] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3344.823245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3345.096368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3349.235846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3352.536645] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3361.587687] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3370.822340] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3373.674293] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3379.519571] Lustre: server umount lustre-OST0000 complete [ 3385.322743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3385.330105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3385.349744] Lustre: Skipped 1 previous similar message [ 3390.443747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3390.450466] Lustre: Skipped 2 previous similar messages [ 3393.721580] Lustre: server umount lustre-MDT0000 complete [ 3395.554070] LustreError: 111014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3395.596392] LustreError: 111014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3396.579748] Lustre: 110323:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451404/real 1788451404] req@00000000b5cca7c5 x1875327126111744/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788451420 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3396.622103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3398.485435] LustreError: 113547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451422 with bad export cookie 986096021050517408 [ 3398.489580] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3398.510794] LustreError: 113547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3398.920492] Lustre: server umount lustre-MDT0001 complete [ 3411.089434] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 3414.343282] Key type lgssc unregistered [ 3414.709497] LNet: 114652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3414.719971] LNetError: 114652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3414.758126] LNet: Removed LNI 192.168.202.141@tcp [ 3415.942955] Key type .llcrypt unregistered [ 3415.947368] Key type ._llcrypt unregistered [ 3436.790142] Key type ._llcrypt registered [ 3436.791849] Key type .llcrypt registered [ 3436.963685] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3438.153447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3438.199051] alg: No test for adler32 (adler32-zlib) [ 3439.515328] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 3439.939174] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 3441.776253] Key type lgssc registered [ 3443.331669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3452.565864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3452.579347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3454.186179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3459.779652] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3464.684739] LustreError: 115882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3464.716658] LustreError: 115882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3469.706431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.794431] LustreError: 115883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.335029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3475.901706] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3482.669083] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3488.815469] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3495.441037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3495.830030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3498.948865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3503.133597] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3512.967346] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3526.114684] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3526.117078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3526.156393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3530.720910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3530.739310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3530.764780] Lustre: Skipped 1 previous similar message [ 3532.137779] Lustre: server umount lustre-OST0000 complete [ 3536.865136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3536.879407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3536.892378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3540.965256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3540.967390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3540.986838] Lustre: Skipped 1 previous similar message [ 3546.092236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3546.104875] Lustre: Skipped 1 previous similar message [ 3551.199500] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3551.407949] Lustre: server umount lustre-MDT0000 complete [ 3556.219572] LustreError: 115862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451580 with bad export cookie 15603220676500500966 [ 3556.226903] LustreError: 115862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3556.227048] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3556.672413] Lustre: server umount lustre-MDT0001 complete [ 3568.643923] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 3571.197888] Key type lgssc unregistered [ 3571.513736] LNet: 119436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3571.523028] LNetError: 119436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3571.545482] LNet: Removed LNI 192.168.202.141@tcp [ 3572.476616] Key type .llcrypt unregistered [ 3572.479271] Key type ._llcrypt unregistered [ 3600.634289] Key type ._llcrypt registered [ 3600.638712] Key type .llcrypt registered [ 3600.753804] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3602.247948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3602.299796] alg: No test for adler32 (adler32-zlib) [ 3603.569877] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 3603.900119] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 3605.615173] Key type lgssc registered [ 3606.880334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3613.825817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3613.838357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3615.263994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.805480] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3625.440820] LustreError: 120851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3625.469625] LustreError: 120851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3627.717432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.944656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3632.261678] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3642.262898] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3648.311139] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3656.776743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3657.180805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3660.288139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3664.355953] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3672.695953] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3679.688588] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3683.666910] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3683.673195] Lustre: Skipped 2 previous similar messages [ 3688.048306] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3688.094364] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3689.008756] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3689.012419] Lustre: Skipped 2 previous similar messages [ 3696.741451] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3696.748607] Lustre: Skipped 1 previous similar message [ 3702.272377] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.41@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3702.285164] Lustre: Skipped 2 previous similar messages [ 3702.544823] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.41@tcp 192.168.0.[2,10]@tcp [ 3708.496298] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.41/24@tcp in log lustre-MDT0000 [ 3708.502163] Lustre: Skipped 2 previous similar messages [ 3714.117742] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.41/24@tcp [ 3714.128222] Lustre: Skipped 1 previous similar message [ 3721.188650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3721.195029] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3721.211212] LustreError: Skipped 1 previous similar message [ 3721.240503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3724.779624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3724.789489] LustreError: Skipped 4 previous similar messages [ 3724.789954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3724.793446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3724.826449] Lustre: Skipped 1 previous similar message [ 3725.933991] Lustre: server umount lustre-OST0000 complete [ 3729.889441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3729.898121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3729.906880] Lustre: Skipped 1 previous similar message [ 3729.919244] Lustre: Skipped 1 previous similar message [ 3735.009144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3735.455387] Lustre: server umount lustre-MDT0000 complete [ 3739.245795] LustreError: 120832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451763 with bad export cookie 15761455275156472993 [ 3739.250625] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3739.256656] LustreError: 120832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3739.626518] Lustre: server umount lustre-MDT0001 complete [ 3751.043799] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 3755.445747] Key type lgssc unregistered [ 3755.867635] LNet: 125746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3755.893403] LNetError: 125746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3756.903497] LNet: Removed LNI 192.168.202.141@tcp [ 3758.193372] Key type .llcrypt unregistered [ 3758.202284] Key type ._llcrypt unregistered [ 3790.119433] Key type ._llcrypt registered [ 3790.126693] Key type .llcrypt registered [ 3790.254225] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3791.437992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3791.562157] alg: No test for adler32 (adler32-zlib) [ 3792.674656] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 3793.040620] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 3794.719270] Key type lgssc registered [ 3795.988178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3804.036741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3804.048971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3805.361801] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3805.381990] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.41/24@tcp [ 3805.399611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3810.377972] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3814.884794] LustreError: 127306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3814.907794] LustreError: 127306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3819.250517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.520812] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3819.558649] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.41/24@tcp [ 3819.606124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3824.876625] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3833.399785] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3840.173676] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3847.436730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3847.930980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3849.995927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3854.956284] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3862.376108] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3874.790535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3874.804120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3874.825280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3874.832550] Lustre: Skipped 1 previous similar message [ 3879.496866] Lustre: server umount lustre-OST0000 complete [ 3885.025943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.030558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3885.043155] Lustre: Skipped 2 previous similar messages [ 3885.053175] Lustre: Skipped 1 previous similar message [ 3890.162835] LustreError: 127306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3890.185223] LustreError: 127306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3890.247870] Lustre: server umount lustre-MDT0000 complete [ 3894.245936] LustreError: 127287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451918 with bad export cookie 2198181732440198050 [ 3894.256905] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3894.267592] LustreError: 127287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3894.789346] Lustre: server umount lustre-MDT0001 complete [ 3906.297296] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 3909.256262] Key type lgssc unregistered [ 3909.598768] LNet: 130894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3909.605227] LNetError: 130894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3909.629191] LNet: Removed LNI 192.168.202.141@tcp [ 3910.615290] Key type .llcrypt unregistered [ 3910.617201] Key type ._llcrypt unregistered [ 3938.443637] Key type ._llcrypt registered [ 3938.445561] Key type .llcrypt registered [ 3938.611175] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 3955.851660] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3957.125833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3957.265729] alg: No test for adler32 (adler32-zlib) [ 3958.330446] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 3958.605185] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 3960.289267] Key type lgssc registered [ 3961.874684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3996.454763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3996.475764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3997.751704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3997.786970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3997.857706] Lustre: lustre-MDT0000: new disk, initializing [ 3997.925180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3997.939748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4002.432256] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4011.409327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4011.540765] Lustre: 134292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4011.595451] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4011.600132] Lustre: Skipped 1 previous similar message [ 4011.682457] Lustre: lustre-MDT0001: new disk, initializing [ 4011.777422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4011.810292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4011.833941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4015.730608] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4024.942716] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4030.401710] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4037.784063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4038.122447] Lustre: lustre-OST0000: new disk, initializing [ 4038.128624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4038.138703] Lustre: 136261:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4038.254753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4044.707657] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4046.433644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4046.445184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4046.548483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4054.436304] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4059.620687] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4059.769351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4064.708812] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4064.986512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4067.304201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4067.305026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4067.339649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4067.344648] Lustre: Skipped 1 previous similar message [ 4072.422611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4072.432146] Lustre: Skipped 1 previous similar message [ 4073.275946] Lustre: server umount lustre-OST0000 complete [ 4077.537133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4077.549319] Lustre: Skipped 1 previous similar message [ 4077.558066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4077.566859] Lustre: Skipped 1 previous similar message [ 4082.656589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4082.663198] Lustre: Skipped 1 previous similar message [ 4087.778731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4087.792652] Lustre: Skipped 1 previous similar message [ 4091.361382] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4091.621423] Lustre: server umount lustre-MDT0000 complete [ 4094.921723] LustreError: 134284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452119 with bad export cookie 18217047248688155559 [ 4094.933265] LustreError: 134284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4094.939633] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4095.218855] Lustre: server umount lustre-MDT0001 complete [ 4104.203358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4104.534708] LustreError: 138481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4104.553898] LustreError: 138481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4104.605807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4109.314464] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4109.794593] LustreError: 138482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4113.893354] LustreError: 138481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4117.884567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4118.334758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4121.911611] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4128.511672] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4133.833553] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4140.570050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4140.857133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4148.492919] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4157.176252] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4162.359429] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4162.573909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4167.292600] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4167.507632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4175.994244] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4176.161416] Lustre: lustre-OST0001: new disk, initializing [ 4176.163948] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4176.167945] Lustre: 141852:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4176.274449] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4182.110831] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4184.184902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4184.197866] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4184.316785] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4191.325386] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4191.578115] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4196.095233] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4196.277104] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4201.810651] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff955592ae5000.ost_server_uuid 50 [ 4203.107703] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff955592ae5000.ost_server_uuid in IDLE state after 0 sec [ 4210.144072] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4210.154274] LustreError: Skipped 1 previous similar message [ 4210.155501] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4210.166871] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4210.171190] Lustre: Skipped 2 previous similar messages [ 4210.181295] Lustre: Skipped 1 previous similar message [ 4216.352759] Lustre: server umount lustre-OST0001 complete [ 4220.390119] LustreError: 140303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4220.416736] LustreError: 140303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4224.992914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4225.008932] LustreError: Skipped 1 previous similar message [ 4225.016683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.512479] LustreError: 140305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4225.536099] LustreError: 140305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4230.636436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4230.644811] Lustre: Skipped 6 previous similar messages [ 4235.231520] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4235.470628] Lustre: server umount lustre-OST0000 complete [ 4240.876757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4240.906118] Lustre: Skipped 1 previous similar message [ 4245.582447] Lustre: server umount lustre-MDT0000 complete [ 4245.985598] LustreError: 139209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4245.998876] LustreError: 139209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4249.217199] LustreError: 138462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452273 with bad export cookie 18217047248688156441 [ 4249.222664] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4249.228126] LustreError: 138462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4249.504100] Lustre: server umount lustre-MDT0001 complete [ 4265.775797] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 4269.866953] Key type lgssc unregistered [ 4270.217859] LNet: 144852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4270.233157] LNetError: 144852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4270.266230] LNet: Removed LNI 192.168.202.141@tcp [ 4271.293915] Key type .llcrypt unregistered [ 4271.296357] Key type ._llcrypt unregistered [ 4310.610759] Key type ._llcrypt registered [ 4310.613535] Key type .llcrypt registered [ 4310.727885] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 4324.749387] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 4325.983468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4326.284567] alg: No test for adler32 (adler32-zlib) [ 4327.369721] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 4327.578828] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 4329.239885] Key type lgssc registered [ 4330.302459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4359.781056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4359.799935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4361.003319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4361.029042] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4361.099405] Lustre: lustre-MDT0000: new disk, initializing [ 4361.186231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4361.212052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4364.848465] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4377.106880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.315633] Lustre: 148999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4377.409558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4377.420119] Lustre: Skipped 1 previous similar message [ 4377.562504] Lustre: lustre-MDT0001: new disk, initializing [ 4377.698992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4377.744642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4377.755216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4382.046692] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4391.976837] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4397.872933] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4405.828598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4406.214675] Lustre: lustre-OST0000: new disk, initializing [ 4406.222072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4406.229162] Lustre: 150969:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4406.332258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4412.133362] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4415.092607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4415.125316] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4415.225028] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4420.771220] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4425.756824] Lustre: Failing over lustre-OST0000 [ 4425.863473] Lustre: server umount lustre-OST0000 complete [ 4443.104117] Lustre: 147227:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452451/real 1788452451] req@00000000be4ddd36 x1875328212435584/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788452467 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4443.118670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4445.804872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4446.382022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.430289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4446.450335] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4447.200085] Lustre: 147226:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452455/real 1788452455] req@000000001fe07cfe x1875328212436096/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788452471 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4448.432775] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4452.259568] Lustre: 147225:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452460/real 1788452460] req@00000000558d2eea x1875328212437888/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788452476 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4452.283609] Lustre: 147225:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4454.082189] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4455.456431] Lustre: lustre-OST0000: Recovery over after 0:07, of 3 clients 3 recovered and 0 were evicted. [ 4455.461080] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4457.569575] Lustre: 147227:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452465/real 1788452465] req@0000000032149ee3 x1875328212439040/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788452481 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4457.614923] Lustre: 147227:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4458.281700] Lustre: Failing over lustre-MDT0000 [ 4458.583378] Lustre: server umount lustre-MDT0000 complete [ 4458.979022] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4459.001595] Lustre: Skipped 2 previous similar messages [ 4464.099842] LustreError: 149012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4464.116764] LustreError: 149012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4465.667860] LustreError: 149008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4469.220920] LustreError: 149006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4469.242068] LustreError: 149006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4474.342618] LustreError: 149011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4474.373370] LustreError: 149011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4477.240944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4477.431347] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4477.650255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4477.710367] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4480.827520] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4482.420813] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4483.058483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4483.066129] Lustre: Skipped 1 previous similar message [ 4483.094342] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4483.146977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4491.745399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4491.756715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4491.767667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4493.288364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4493.304017] Lustre: Skipped 1 previous similar message [ 4496.681976] Lustre: server umount lustre-OST0000 complete [ 4503.522791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4503.544908] Lustre: Skipped 2 previous similar messages [ 4503.552666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4503.552884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4503.579847] Lustre: Skipped 1 previous similar message [ 4506.956696] Lustre: server umount lustre-MDT0000 complete [ 4508.640323] LustreError: 149012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4508.662658] LustreError: 149012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4511.347795] LustreError: 152230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452535 with bad export cookie 11293927035144402797 [ 4511.352135] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4511.363136] LustreError: 152230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4511.769247] Lustre: server umount lustre-MDT0001 complete [ 4522.949734] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 4526.079882] Key type lgssc unregistered [ 4526.444877] LNet: 154626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4526.456368] LNetError: 154626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4526.469886] LNet: Removed LNI 192.168.202.141@tcp [ 4527.412587] Key type .llcrypt unregistered [ 4527.416943] Key type ._llcrypt unregistered [ 4565.360503] Key type ._llcrypt registered [ 4565.362147] Key type .llcrypt registered [ 4565.500196] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 4566.875811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4566.957467] alg: No test for adler32 (adler32-zlib) [ 4567.974541] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 4568.199800] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 4569.855211] Key type lgssc registered [ 4570.904869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4577.862601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4577.874646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4579.293076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4584.248812] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4589.537112] LustreError: 156145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4589.584653] LustreError: 156145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4592.941941] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4594.659173] LustreError: 156146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4596.764118] LustreError: 156174:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4596.773525] LustreError: 156174:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4596.782659] LustreError: 156174:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4597.125294] Lustre: server umount lustre-MDT0000 complete [ 4616.713509] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 4629.321879] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 4671.021529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4671.181812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4671.209221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4671.327403] Lustre: lustre-MDT0000: new disk, initializing [ 4671.427955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.459222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4677.158791] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4687.304149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4687.433749] Lustre: 159480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4687.480787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4687.483935] Lustre: Skipped 1 previous similar message [ 4687.545440] Lustre: lustre-MDT0001: new disk, initializing [ 4687.619821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4687.648398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4687.661900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4693.214940] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4703.694044] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4710.668783] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4718.319021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4718.624800] Lustre: lustre-OST0000: new disk, initializing [ 4718.629085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4718.636138] Lustre: 161449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4718.740551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4725.335872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4725.355321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4725.526475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4725.942907] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4736.039581] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4740.605284] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4742.301682] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 12:26:05 (1788452765) [ 4945.891781] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4945.893923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4945.914758] Lustre: Skipped 2 previous similar messages [ 4945.935090] Lustre: Skipped 2 previous similar messages [ 4951.012659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4951.038940] Lustre: Skipped 2 previous similar messages [ 4952.128990] Lustre: server umount lustre-MDT0000 complete [ 4956.114096] LustreError: 159473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452980 with bad export cookie 11100146596932826721 [ 4956.130426] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4956.143676] LustreError: 159473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4956.144100] LustreError: 163214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4956.144546] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4956.145098] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4961.250385] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4961.254861] LustreError: 159488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4962.800856] Lustre: server umount lustre-MDT0001 complete [ 4981.215129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4981.443771] Lustre: server umount lustre-OST0000 complete [ 4990.558900] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 4999.414238] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 5039.454436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5039.861786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5039.877735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5039.956506] Lustre: lustre-MDT0000: new disk, initializing [ 5040.002933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5040.023656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5045.690239] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5057.396782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.512860] Lustre: 166114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5057.560551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5057.564942] Lustre: Skipped 1 previous similar message [ 5057.643549] Lustre: lustre-MDT0001: new disk, initializing [ 5057.781849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5057.853879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5057.867797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5062.251258] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5072.148539] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5077.957248] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5084.826938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5085.214371] Lustre: lustre-OST0000: new disk, initializing [ 5085.223339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5085.234465] Lustre: 168082:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5085.360213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5091.621669] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5091.980618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5091.985201] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5092.192341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5101.204607] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5105.774359] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5105.977575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5110.173136] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5110.411148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5112.802542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5112.812694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5112.817868] LustreError: Skipped 1 previous similar message [ 5112.834206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5118.351712] Lustre: server umount lustre-OST0000 complete [ 5123.048203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5123.056564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5123.059114] Lustre: Skipped 2 previous similar messages [ 5123.071746] Lustre: Skipped 4 previous similar messages [ 5136.351307] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5136.567267] Lustre: server umount lustre-MDT0000 complete [ 5138.401653] LustreError: 166121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5138.411316] LustreError: 166121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5139.555398] LustreError: 166106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453163 with bad export cookie 11100146596932985551 [ 5139.560640] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5139.570957] LustreError: 166106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5139.844407] Lustre: server umount lustre-MDT0001 complete [ 5147.000968] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:32:49 (1788453169) [ 5163.758731] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 5175.492946] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 5208.173317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5208.429810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5208.452536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5208.570628] Lustre: lustre-MDT0000: new disk, initializing [ 5208.712168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5208.728044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5213.266460] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5222.817047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5223.006344] Lustre: 172837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5223.410830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5227.543840] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5238.285968] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5244.913586] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5251.070420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5251.388689] Lustre: lustre-OST0000: new disk, initializing [ 5251.391934] Lustre: Skipped 1 previous similar message [ 5251.399905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5251.407515] Lustre: Skipped 2 previous similar messages [ 5251.425735] Lustre: 174814:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5257.837239] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5258.348643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5258.353457] Lustre: Skipped 1 previous similar message [ 5258.361375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5258.411551] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5269.612239] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5279.226215] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5279.226965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5279.234169] LustreError: Skipped 1 previous similar message [ 5279.239311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5279.244724] Lustre: Skipped 1 previous similar message [ 5279.257320] Lustre: Skipped 5 previous similar messages [ 5283.273286] Lustre: server umount lustre-OST0000 complete [ 5289.444090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5289.452586] Lustre: Skipped 1 previous similar message [ 5292.474980] Lustre: server umount lustre-MDT0000 complete [ 5294.562082] LustreError: 172844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5294.585459] LustreError: 172844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5295.758770] LustreError: 173637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453319 with bad export cookie 11100146596932986433 [ 5295.762557] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5295.765523] LustreError: 173637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5296.014613] Lustre: server umount lustre-MDT0001 complete [ 5303.722898] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:35:26 (1788453326) [ 5319.450668] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 5329.122606] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 5364.318421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5364.543701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5364.565488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5364.648731] Lustre: lustre-MDT0000: new disk, initializing [ 5364.717103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5364.721725] Lustre: Skipped 2 previous similar messages [ 5364.738306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5369.210746] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5379.179565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5379.346783] Lustre: 179136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5379.670182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5384.501961] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5393.401198] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5397.928969] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5404.432348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5404.620878] Lustre: 181112:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5411.770055] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5414.452233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5414.554288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5422.444244] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5430.239875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5430.246252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5430.256378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5430.263469] Lustre: Skipped 1 previous similar message [ 5430.281527] Lustre: Skipped 3 previous similar messages [ 5435.653763] Lustre: server umount lustre-OST0000 complete [ 5445.520724] Lustre: server umount lustre-MDT0000 complete [ 5445.601394] LustreError: 179148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5445.610740] LustreError: 179148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5448.880091] LustreError: 179128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453473 with bad export cookie 11100146596932987518 [ 5448.889235] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5448.900844] LustreError: 179128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5449.259320] Lustre: server umount lustre-MDT0001 complete [ 5459.700249] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 5463.113643] Key type lgssc unregistered [ 5463.558776] LNet: 182967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5463.566908] LNetError: 182967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5464.613712] LNet: Removed LNI 192.168.202.141@tcp [ 5465.485250] Key type .llcrypt unregistered [ 5465.489635] Key type ._llcrypt unregistered [ 5487.816434] Key type ._llcrypt registered [ 5487.820368] Key type .llcrypt registered [ 5487.938219] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 5488.914701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5488.996879] alg: No test for adler32 (adler32-zlib) [ 5489.994307] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 5490.182559] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 5491.823313] Key type lgssc registered [ 5492.866727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5499.930476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5499.946480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5501.280549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5505.416305] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5511.648852] LustreError: 184358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5511.676928] LustreError: 184358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5513.155442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5513.475039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5517.824670] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5524.655896] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5530.543294] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5537.176619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5537.455260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5543.871739] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5546.702754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5550.366943] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5572.583488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5572.597869] Lustre: Skipped 1 previous similar message [ 5572.601407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5572.609290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5572.620749] Lustre: Skipped 1 previous similar message [ 5577.819475] Lustre: server umount lustre-OST0000 complete [ 5582.825022] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5582.828066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5582.833620] Lustre: Skipped 1 previous similar message [ 5582.840202] Lustre: Skipped 1 previous similar message [ 5587.940839] LustreError: 184358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5587.985638] LustreError: 184358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5588.059729] Lustre: server umount lustre-MDT0000 complete [ 5591.711943] LustreError: 185093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453615 with bad export cookie 2428218906004927018 [ 5591.717199] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5591.728746] LustreError: 185093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5592.376609] Lustre: server umount lustre-MDT0001 complete [ 5605.377355] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 5607.872861] Key type lgssc unregistered [ 5608.136297] LNet: 187915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5608.146192] LNetError: 187915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5609.191556] LNet: Removed LNI 192.168.202.141@tcp [ 5609.882779] Key type .llcrypt unregistered [ 5609.884232] Key type ._llcrypt unregistered [ 5638.378935] Key type ._llcrypt registered [ 5638.380588] Key type .llcrypt registered [ 5638.517926] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 5640.261296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5640.539330] alg: No test for adler32 (adler32-zlib) [ 5641.685490] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 5642.004588] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 5643.655657] Key type lgssc registered [ 5645.239928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5653.606461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5653.655209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5655.291604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5658.950505] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5665.762694] LustreError: 189329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5665.789048] LustreError: 189329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5667.730969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5668.173087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5672.471300] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5679.332623] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5685.033369] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5692.262727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5692.576080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5699.333861] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5700.851088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5707.823181] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5715.996229] Lustre: lustre-OST0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 5716.453388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5716.462175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5721.081296] Lustre: lustre-OST0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 5721.085689] Lustre: Skipped 2 previous similar messages [ 5726.202807] Lustre: lustre-OST0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 5726.208147] Lustre: Skipped 2 previous similar messages [ 5729.247136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5729.356575] Lustre: server umount lustre-OST0000 complete [ 5735.979707] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff95559aae7800.ost_server_uuid 50 [ 5737.492811] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5751.611294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5751.628544] Lustre: Skipped 1 previous similar message [ 5751.641089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5751.652092] Lustre: Skipped 3 previous similar messages [ 5756.897388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5756.905753] Lustre: Skipped 1 previous similar message [ 5757.923318] LustreError: 189330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5757.975685] Lustre: server umount lustre-MDT0000 complete [ 5761.643085] LustreError: 189310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453785 with bad export cookie 18171647528222976525 [ 5761.647126] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5761.654338] LustreError: 189310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5762.048825] Lustre: server umount lustre-MDT0001 complete [ 5770.951815] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:43:13 (1788453793) [ 5777.500938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5777.736961] LustreError: 193247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.752301] LustreError: 193247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5777.792213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5781.490906] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5783.009917] LustreError: 193248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5787.117517] LustreError: 193247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5789.528308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5789.806881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5794.705232] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5801.526690] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5807.027896] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5813.855183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5814.379546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5820.673668] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5822.590684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5828.079028] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5834.611951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5834.808634] Lustre: lustre-OST0001: new disk, initializing [ 5834.812718] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5834.818587] Lustre: 196085:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5834.915341] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5841.073935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5841.089477] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5841.228877] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5841.834841] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5851.533574] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5856.743582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5856.748580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5856.768549] Lustre: Skipped 1 previous similar message [ 5856.771864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5860.797177] Lustre: server umount lustre-OST0000 complete [ 5861.861629] LustreError: 195068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5861.892141] LustreError: 195068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5867.578658] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5867.857849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5873.161547] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5873.377293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5882.340597] LustreError: 196094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5882.381654] LustreError: 196094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 5887.457102] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5887.459103] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5887.464195] LustreError: Skipped 1 previous similar message [ 5887.466237] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5887.466244] Lustre: Skipped 2 previous similar messages [ 5887.484892] Lustre: Skipped 2 previous similar messages [ 5893.437358] Lustre: server umount lustre-OST0001 complete [ 5897.200366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5897.204855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5911.007739] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5911.195900] Lustre: server umount lustre-MDT0000 complete [ 5915.229677] LustreError: 193227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453939 with bad export cookie 18171647528222977596 [ 5915.266811] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5915.741122] Lustre: server umount lustre-MDT0001 complete [ 5945.675226] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:46:07 (1788453967) [ 5952.723222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5952.997827] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5953.076107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5953.457893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5958.533669] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5967.964735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5968.147350] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5968.189750] Lustre: 199524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5968.294280] LustreError: 199535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5968.317557] LustreError: 199535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5973.111845] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 5981.346768] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5986.601807] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5993.022973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5993.188383] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6001.315095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6001.475340] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6011.791915] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6020.502992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6020.553636] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6020.917982] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6020.940633] Lustre: Skipped 2 previous similar messages [ 6028.031663] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6035.650443] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6041.092284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6041.099849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6041.127712] Lustre: Skipped 2 previous similar messages [ 6041.156187] Lustre: Skipped 10 previous similar messages [ 6044.358899] Lustre: server umount lustre-OST0000 complete [ 6046.216402] LustreError: 201724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6046.269383] LustreError: 201724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6061.536477] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6061.543374] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6061.555242] LustreError: Skipped 1 previous similar message [ 6061.604800] Lustre: Skipped 1 previous similar message [ 6067.054573] Lustre: server umount lustre-OST0001 complete [ 6071.268560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6077.640886] Lustre: server umount lustre-MDT0000 complete [ 6082.003246] LustreError: 199518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454106 with bad export cookie 18171647528222979003 [ 6082.013464] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6082.025104] LustreError: 199518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6082.748644] Lustre: server umount lustre-MDT0001 complete [ 6107.011894] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:48:49 (1788454129) [ 6122.611309] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 6132.461242] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 6167.578142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6167.718794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6167.739476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6167.789361] Lustre: lustre-MDT0000: new disk, initializing [ 6167.842817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6167.860422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6173.309614] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6184.203554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6184.390460] Lustre: 207623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6184.473588] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6184.485553] Lustre: Skipped 1 previous similar message [ 6184.653848] Lustre: lustre-MDT0001: new disk, initializing [ 6184.886242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6184.889798] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6189.602650] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6198.143680] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6203.699855] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6210.153798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6210.606347] Lustre: lustre-OST0000: new disk, initializing [ 6210.609299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6210.615957] Lustre: 209593:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6216.244903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6216.265287] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6216.392812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6217.130394] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6227.948745] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6233.265769] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6233.506980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6238.156643] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6238.388032] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6242.274127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6242.285837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6242.286488] LustreError: Skipped 1 previous similar message [ 6242.297226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6242.323615] Lustre: Skipped 3 previous similar messages [ 6242.358445] Lustre: Skipped 10 previous similar messages [ 6246.892752] Lustre: server umount lustre-OST0000 complete [ 6257.639596] LustreError: 207634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6257.679715] LustreError: 207634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6257.766403] Lustre: server umount lustre-MDT0000 complete [ 6261.646612] LustreError: 207613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454285 with bad export cookie 18171647528222980564 [ 6261.648284] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6261.653127] LustreError: 207613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6262.047867] Lustre: server umount lustre-MDT0001 complete [ 6271.333838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6276.257341] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6283.845702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6288.970831] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6297.150720] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6303.258732] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6311.064251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6315.687664] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6321.997757] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6326.303121] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6326.538764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6330.876745] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6331.053732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6332.901203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6332.907386] LustreError: Skipped 1 previous similar message [ 6332.910651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6332.919514] Lustre: Skipped 2 previous similar messages [ 6338.542198] Lustre: server umount lustre-OST0000 complete [ 6345.737332] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6350.788427] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6381.910861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6385.328263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.354180] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6390.204603] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6398.794358] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6400.330522] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6400.752660] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.766325] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6406.125485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6406.129562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6406.141931] LustreError: Skipped 2 previous similar messages [ 6406.152848] Lustre: Skipped 7 previous similar messages [ 6410.860343] Lustre: server umount lustre-OST0000 complete [ 6429.151129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6429.439064] Lustre: server umount lustre-MDT0000 complete [ 6433.223393] LustreError: 211792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454457 with bad export cookie 18171647528222981446 [ 6433.229872] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6433.234021] LustreError: 211792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6433.570348] Lustre: server umount lustre-MDT0001 complete [ 6442.663293] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:54:25 (1788454465) [ 6450.275216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6451.123819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6451.126759] Lustre: Skipped 6 previous similar messages [ 6455.915046] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6464.582566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6470.116561] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6477.992861] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6485.850974] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6497.615685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6505.513400] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6513.920125] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6519.061171] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6522.644488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6527.641823] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6527.876868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6534.292375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6534.493734] Lustre: lustre-OST0001: new disk, initializing [ 6534.496750] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6534.501949] Lustre: 221173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6541.417753] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6542.920736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6542.932244] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6543.137901] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6551.594747] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6557.401942] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6557.539793] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6561.936760] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6562.164659] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6568.930747] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6568.948647] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6568.969967] Lustre: Skipped 5 previous similar messages [ 6570.503551] Lustre: server umount lustre-OST0001 complete [ 6574.085195] LustreError: 219603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6574.104941] LustreError: 219603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6575.903754] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6576.093690] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6580.708916] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6580.966782] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6611.413738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6617.063348] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6617.081920] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6617.091768] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6619.798865] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6628.556352] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6630.606937] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6638.669454] Lustre: server umount lustre-OST0001 complete [ 6662.729899] Lustre: server umount lustre-OST0000 complete [ 6669.883614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6669.893934] Lustre: Skipped 12 previous similar messages [ 6676.164144] Lustre: server umount lustre-MDT0000 complete [ 6681.375188] LustreError: 217761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454705 with bad export cookie 18171647528222982412 [ 6681.380684] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6681.387453] LustreError: 217761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6688.018574] Lustre: server umount lustre-MDT0001 complete [ 6712.973650] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:58:56 (1788454736) [ 6719.926943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6720.258897] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6720.306664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6725.887464] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6734.302429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6734.393761] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6734.415518] Lustre: 226415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6738.665813] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6746.814137] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6753.531479] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6762.004485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6762.218355] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6770.353612] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6779.127800] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6790.217142] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6790.349186] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6798.803427] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6806.088505] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6807.970361] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6812.343978] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6812.605775] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6817.463820] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6817.745295] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6824.800424] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff955591c3c000.ost_server_uuid 50 [ 6826.776536] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff955591c3c000.ost_server_uuid in IDLE state after 0 sec [ 6827.992755] Lustre: Permanently deactivating lustre-OST0001 [ 6828.012965] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6833.168216] Lustre: Permanently reactivating lustre-OST0001 [ 6833.180456] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6833.185506] Lustre: Skipped 2 previous similar messages [ 6838.769211] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6838.778119] LustreError: Skipped 4 previous similar messages [ 6838.784550] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6838.792509] Lustre: Skipped 7 previous similar messages [ 6844.737690] Lustre: server umount lustre-OST0001 complete [ 6855.512069] Lustre: server umount lustre-OST0000 complete [ 6865.587327] Lustre: server umount lustre-MDT0000 complete [ 6870.757720] LustreError: 228276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454894 with bad export cookie 18171647528222983693 [ 6870.780295] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6871.471510] Lustre: server umount lustre-MDT0001 complete [ 6901.730303] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 13:02:03 (1788454923) [ 6911.750278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6912.246815] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6912.311563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6912.334651] Lustre: Skipped 2 previous similar messages [ 6918.880227] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6928.587642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6928.762516] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6928.803706] Lustre: 233128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6933.270480] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6941.758991] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6947.214310] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6953.956850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6954.167922] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6954.178313] Lustre: Permanently deactivating lustre-OST0000 [ 6954.184549] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6960.706706] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6960.718205] LustreError: 235298:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 6961.817907] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6970.537844] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6977.926582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6978.000031] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6978.304577] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6978.309744] Lustre: Skipped 11 previous similar messages [ 6986.221100] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6988.501893] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6996.201582] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6998.336988] Lustre: Permanently reactivating lustre-OST0000 [ 6998.359179] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6998.364588] Lustre: Skipped 2 previous similar messages [ 7005.349056] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7005.360761] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 45 seconds. I think it's dead, and I am evicting it. exp 00000000fc9aca6b, cur 1788455029 deadline 1788455014 last 1788454984 [ 7005.364443] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7005.398760] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7005.409320] Lustre: Skipped 1 previous similar message [ 7005.479741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7028.191564] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7028.468709] Lustre: server umount lustre-OST0001 complete [ 7038.819335] Lustre: server umount lustre-OST0000 complete [ 7052.337969] Lustre: server umount lustre-MDT0000 complete [ 7056.952735] LustreError: 233121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455081 with bad export cookie 18171647528222985366 [ 7056.965728] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7056.967668] LustreError: 233121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7057.362808] Lustre: server umount lustre-MDT0001 complete [ 7072.411682] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 7076.532235] Key type lgssc unregistered [ 7076.958737] LNet: 238014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7076.976481] LNetError: 238014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7076.992827] LNet: Removed LNI 192.168.202.141@tcp [ 7078.318349] Key type .llcrypt unregistered [ 7078.322410] Key type ._llcrypt unregistered [ 7106.285899] Key type ._llcrypt registered [ 7106.291991] Key type .llcrypt registered [ 7106.385779] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7107.409982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7107.438019] alg: No test for adler32 (adler32-zlib) [ 7108.675985] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 7109.101880] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 7110.919289] Key type lgssc registered [ 7111.765514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7123.890629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7123.937973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7125.692666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7132.358499] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7135.206487] LustreError: 239478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7135.220246] LustreError: 239478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7140.320761] LustreError: 239479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7142.377820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7142.463985] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7142.502797] Lustre: 239465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7142.511759] Lustre: Permanently deactivating lustre-MDT0001 [ 7142.524780] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7142.692212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7146.760543] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7148.112118] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7156.951444] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7166.829394] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7175.699225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7176.152856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7183.564814] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7187.468204] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7187.633782] LustreError: 241323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7192.547412] LustreError: 241325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7193.071924] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7195.658623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7200.742660] LustreError: 241326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7200.784129] LustreError: 241326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7201.818259] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7202.046481] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7207.427730] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7207.463529] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7210.130786] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7219.323491] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7222.041506] Lustre: Permanently reactivating lustre-MDT0001 [ 7222.055328] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7222.058295] Lustre: Skipped 2 previous similar messages [ 7226.406847] Lustre: lustre-MDT0001: Client e107e4e6-6941-4491-b591-6bde6a813276 (at 192.168.202.41@tcp) reconnecting [ 7228.672726] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7228.701957] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7228.715179] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7228.734605] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7232.586892] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7234.326989] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7240.419309] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7240.687782] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7246.038212] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7246.342109] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7248.712493] Lustre: Permanently deactivating lustre-MDT0001 [ 7248.728449] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7248.736072] Lustre: Skipped 2 previous similar messages [ 7257.759309] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7270.918661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7270.933572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7270.952496] LustreError: 239479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7270.996245] Lustre: server umount lustre-MDT0000 complete [ 7274.710818] LustreError: 239458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455298 with bad export cookie 2647040590624674887 [ 7274.712952] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7274.720385] LustreError: 239458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7275.162294] Lustre: server umount lustre-MDT0001 complete [ 7282.313508] Lustre: server umount lustre-OST0000 complete [ 7295.906804] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 13:08:38 (1788455318) [ 7307.198923] Lustre: server umount lustre-OST0001 complete [ 7316.376703] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 7329.911255] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7366.460943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7367.128613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7367.139544] Lustre: Skipped 2 previous similar messages [ 7367.212864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7367.449326] Lustre: lustre-MDT0000: new disk, initializing [ 7367.644638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7367.691553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7375.678875] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7390.547974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.742874] Lustre: 247795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7390.809751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7390.816493] Lustre: Skipped 1 previous similar message [ 7390.890505] Lustre: lustre-MDT0001: new disk, initializing [ 7390.953654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7390.978726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7390.988909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7393.884597] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7402.804558] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7408.825972] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7416.292263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7416.607946] Lustre: lustre-OST0000: new disk, initializing [ 7416.617272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7416.623432] Lustre: 249763:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7416.716415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7423.355493] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7426.698814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7426.720658] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7426.889803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7433.120908] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7435.162668] LustreError: 249327:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7444.589827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7444.840835] Lustre: lustre-OST0001: new disk, initializing [ 7444.843046] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7444.848847] Lustre: 250876:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7445.175912] LustreError: 249327:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7445.232932] LustreError: 247801:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7450.425884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7450.455324] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7450.656278] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7452.177373] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7455.319147] LustreError: 247801:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7461.376534] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7463.483654] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7465.974246] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7465.977681] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7465.986244] Lustre: Skipped 3 previous similar messages [ 7465.994870] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7471.082558] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7471.085364] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7471.090907] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7471.096828] Lustre: Skipped 2 previous similar messages [ 7476.203438] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7476.213498] Lustre: Skipped 2 previous similar messages [ 7480.288656] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7480.559482] Lustre: server umount lustre-OST0001 complete [ 7481.325230] LustreError: 249749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7481.334678] LustreError: 249749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7498.722163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7498.737304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7498.737996] LustreError: Skipped 1 previous similar message [ 7498.740316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7498.774247] Lustre: Skipped 1 previous similar message [ 7498.805974] Lustre: Skipped 2 previous similar messages [ 7504.725913] Lustre: server umount lustre-OST0000 complete [ 7508.961400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7508.970167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7508.981510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7508.990437] Lustre: Skipped 2 previous similar messages [ 7517.153172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7517.160634] Lustre: Skipped 3 previous similar messages [ 7522.271673] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7522.508681] Lustre: server umount lustre-MDT0000 complete [ 7525.834179] LustreError: 247787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455550 with bad export cookie 2647040590624677925 [ 7525.839897] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7525.840358] LustreError: 247787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7540.191192] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7540.392670] Lustre: server umount lustre-MDT0001 complete [ 7553.007237] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 7555.679845] Key type lgssc unregistered [ 7555.923740] LNet: 252913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7555.936738] LNetError: 252913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7555.967692] LNet: Removed LNI 192.168.202.141@tcp [ 7557.298241] Key type .llcrypt unregistered [ 7557.307683] Key type ._llcrypt unregistered [ 7602.851800] Key type ._llcrypt registered [ 7602.855181] Key type .llcrypt registered [ 7602.970715] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7604.714055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7604.743068] alg: No test for adler32 (adler32-zlib) [ 7606.075878] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 7606.308995] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 7608.031519] Key type lgssc registered [ 7609.282743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7619.188445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7619.220910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7620.879063] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7620.952573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7621.230138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7627.651689] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7637.387828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7637.471903] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7637.494712] Lustre: 255121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7637.882789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7644.062109] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7653.186573] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7660.286238] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7667.192214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7667.395910] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7667.703041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7674.112969] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7675.645840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7684.011457] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7742.434271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7742.434731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.444562] LustreError: Skipped 1 previous similar message [ 7742.449713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7742.451829] Lustre: Skipped 1 previous similar message [ 7742.724298] Lustre: server umount lustre-OST0000 complete [ 7749.237041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7763.245980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7763.526655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7768.826716] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7776.285551] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7781.870600] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7781.901134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7781.902629] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7817.697489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7817.702801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7817.713671] LustreError: Skipped 1 previous similar message [ 7817.714592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7817.714599] Lustre: Skipped 1 previous similar message [ 7817.751741] Lustre: Skipped 1 previous similar message [ 7821.844888] Lustre: server umount lustre-OST0000 complete [ 7827.426423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7827.444264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7827.475535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7827.488981] Lustre: Skipped 1 previous similar message [ 7832.447398] Lustre: server umount lustre-MDT0000 complete [ 7837.147034] LustreError: 255114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455861 with bad export cookie 5908080934467071731 [ 7837.158713] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7837.164038] LustreError: 255114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7837.630417] Lustre: server umount lustre-MDT0001 complete [ 7850.771571] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 7854.030527] Key type lgssc unregistered [ 7854.439995] LNet: 260545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7854.455170] LNetError: 260545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7854.476403] LNet: Removed LNI 192.168.202.141@tcp [ 7855.411793] Key type .llcrypt unregistered [ 7855.416897] Key type ._llcrypt unregistered [ 7886.706790] Key type ._llcrypt registered [ 7886.709270] Key type .llcrypt registered [ 7886.852848] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7888.232902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7888.345534] alg: No test for adler32 (adler32-zlib) [ 7890.236970] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 7890.504709] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 7892.185405] Key type lgssc registered [ 7894.523863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7905.442057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7905.466454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7907.098897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7913.859345] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7917.545945] LustreError: 261959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7917.601857] LustreError: 261959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7922.662125] LustreError: 261960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7927.781329] LustreError: 261959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7930.557234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7931.295227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7938.998914] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7947.965616] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7954.395273] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7962.863245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7963.267651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7972.294974] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7978.671736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7984.269602] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8017.899958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8017.905817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8017.917363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8019.955241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8019.959685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8019.965256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8020.012835] Lustre: Skipped 1 previous similar message [ 8021.686375] Lustre: server umount lustre-OST0000 complete [ 8028.646642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8028.682535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8028.719521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8032.161520] Lustre: server umount lustre-MDT0000 complete [ 8035.300087] LustreError: 261959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8035.337751] LustreError: 261959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8037.765604] LustreError: 261939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456061 with bad export cookie 9103032319243995218 [ 8037.772105] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8037.773481] LustreError: 261939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8038.066117] Lustre: server umount lustre-MDT0001 complete [ 8060.061967] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 8064.568517] Key type lgssc unregistered [ 8065.054039] LNet: 266294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8065.063721] LNetError: 266294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8065.118570] LNet: Removed LNI 192.168.202.141@tcp [ 8066.583225] Key type .llcrypt unregistered [ 8066.585434] Key type ._llcrypt unregistered [ 8087.929492] Key type ._llcrypt registered [ 8087.942540] Key type .llcrypt registered [ 8088.084284] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8089.596400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8089.678166] alg: No test for adler32 (adler32-zlib) [ 8090.822966] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 8091.174524] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 8092.903288] Key type lgssc registered [ 8094.151759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8102.535166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8102.578625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8104.147440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8109.662250] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8114.659318] LustreError: 267513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.672567] LustreError: 267513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8119.099878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8119.441381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8124.366632] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8133.517675] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8140.717665] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8148.697753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8148.944346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8150.049419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8155.928947] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8166.982376] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8185.824518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8185.827823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8185.839178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8189.930485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8189.932975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8189.934373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8190.005016] Lustre: Skipped 1 previous similar message [ 8191.487644] Lustre: server umount lustre-OST0000 complete [ 8200.173185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8200.188734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8200.200604] Lustre: Skipped 1 previous similar message [ 8200.233045] Lustre: Skipped 1 previous similar message [ 8203.412638] Lustre: server umount lustre-MDT0000 complete [ 8205.285856] LustreError: 267512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8205.303453] LustreError: 267512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8207.748080] LustreError: 267494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456231 with bad export cookie 17543522292804804581 [ 8207.754231] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8207.774355] LustreError: 267494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8208.519559] Lustre: server umount lustre-MDT0001 complete [ 8221.296222] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 8225.027825] Key type lgssc unregistered [ 8225.436994] LNet: 271191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8225.443881] LNetError: 271191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8225.479817] LNet: Removed LNI 192.168.202.141@tcp [ 8226.661205] Key type .llcrypt unregistered [ 8226.669727] Key type ._llcrypt unregistered [ 8256.626964] Key type ._llcrypt registered [ 8256.630380] Key type .llcrypt registered [ 8256.785584] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8258.089108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8258.188329] alg: No test for adler32 (adler32-zlib) [ 8259.319732] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 8259.499831] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 8261.487241] Key type lgssc registered [ 8263.023343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8273.903876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8273.933387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8275.709871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8282.036093] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8285.159315] LustreError: 272609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8285.190245] LustreError: 272609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8290.276960] LustreError: 272610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8293.968933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8294.721029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8300.818041] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8310.804263] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8319.854717] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8328.158374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8328.682587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8336.965619] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8345.085071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8345.814237] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8374.755327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8374.771710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.788941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8375.782802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8375.784913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8375.797957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8375.807128] Lustre: Skipped 1 previous similar message [ 8379.232360] Lustre: server umount lustre-OST0000 complete [ 8386.022833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8386.034461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8386.048882] Lustre: Skipped 1 previous similar message [ 8386.063921] Lustre: Skipped 1 previous similar message [ 8390.600286] Lustre: server umount lustre-MDT0000 complete [ 8391.137544] LustreError: 273852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8391.158301] LustreError: 273852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8396.137419] LustreError: 272591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456420 with bad export cookie 2079130568405600863 [ 8396.151053] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8396.167920] LustreError: 272591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8396.781113] Lustre: server umount lustre-MDT0001 complete [ 8412.127243] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 8416.438387] Key type lgssc unregistered [ 8416.919327] LNet: 276920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8416.948933] LNetError: 276920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8418.045349] LNet: Removed LNI 192.168.202.141@tcp [ 8420.023952] Key type .llcrypt unregistered [ 8420.030248] Key type ._llcrypt unregistered [ 8446.160673] Key type ._llcrypt registered [ 8446.166029] Key type .llcrypt registered [ 8446.267194] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8448.056161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8448.111314] alg: No test for adler32 (adler32-zlib) [ 8449.375901] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 8449.813052] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 8451.589097] Key type lgssc registered [ 8453.085428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8463.133779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8463.152609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8464.914063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8471.593778] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8475.105723] LustreError: 278156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8475.126626] LustreError: 278156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8476.134225] LustreError: 278158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8481.252866] LustreError: 278160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8482.919791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8483.638193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8490.007846] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8500.295261] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8507.350310] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8515.123958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8515.711606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8522.356900] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8533.418882] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8536.288393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8557.028836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8557.032556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8557.040364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8560.903653] Lustre: server umount lustre-OST0000 complete [ 8565.740045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8565.746906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8565.757235] Lustre: Skipped 1 previous similar message [ 8565.761929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8565.764438] Lustre: Skipped 1 previous similar message [ 8567.274438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8567.287368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8572.022859] Lustre: server umount lustre-MDT0000 complete [ 8572.386288] LustreError: 278163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8572.398433] LustreError: 278163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8575.929796] LustreError: 278122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456600 with bad export cookie 10849585415917044407 [ 8575.956714] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8576.447862] Lustre: server umount lustre-MDT0001 complete [ 8587.945944] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 8593.464421] Key type lgssc unregistered [ 8593.899778] LNet: 281861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8593.914734] LNetError: 281861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8593.936654] LNet: Removed LNI 192.168.202.141@tcp [ 8595.559126] Key type .llcrypt unregistered [ 8595.560828] Key type ._llcrypt unregistered [ 8611.575569] Key type ._llcrypt registered [ 8611.578693] Key type .llcrypt registered [ 8611.724957] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8629.508137] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 8649.905900] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8650.911828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8650.974979] alg: No test for adler32 (adler32-zlib) [ 8652.185901] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 8652.526218] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 8654.255854] Key type lgssc registered [ 8655.499395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8697.241581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8697.276960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8698.635884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8698.694174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8698.790844] Lustre: lustre-MDT0000: new disk, initializing [ 8698.895131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8698.959560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8704.404675] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8716.852026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8717.069302] Lustre: 285410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8717.185356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8717.190890] Lustre: Skipped 1 previous similar message [ 8717.265638] Lustre: lustre-MDT0001: new disk, initializing [ 8717.387443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8717.419571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8717.434101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8724.329312] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8736.864520] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8742.608768] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8748.353102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8748.640398] Lustre: lustre-OST0000: new disk, initializing [ 8748.643674] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8748.653284] Lustre: 287381:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8748.764106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8754.282440] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8758.318681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8758.339435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8758.428354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8764.289450] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8770.699254] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8770.909186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8775.177249] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8775.465241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8778.729380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8778.763267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8778.779911] Lustre: Skipped 1 previous similar message [ 8783.840640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8783.861383] Lustre: Skipped 1 previous similar message [ 8784.465139] Lustre: server umount lustre-OST0000 complete [ 8794.085454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.087855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8794.109492] Lustre: Skipped 1 previous similar message [ 8794.113578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8796.067683] Lustre: server umount lustre-MDT0000 complete [ 8801.501434] LustreError: 285404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456825 with bad export cookie 12871223836274830117 [ 8801.508331] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8801.515274] LustreError: 285404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8802.066965] Lustre: server umount lustre-MDT0001 complete [ 8812.023797] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:33:54 (1788456834) [ 8820.397705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8820.657128] LustreError: 289791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8820.674460] LustreError: 289791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8820.705654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8825.836121] LustreError: 289792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8827.497813] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8830.013269] LustreError: 289791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8835.043124] LustreError: 289792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8839.328644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8840.450353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8848.660955] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8863.694138] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8871.189210] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8881.709378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8882.171285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8888.410662] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8899.512645] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8910.826788] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8910.845552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8910.845659] LustreError: Skipped 1 previous similar message [ 8910.849543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8910.849554] Lustre: Skipped 1 previous similar message [ 8910.878057] Lustre: Skipped 2 previous similar messages [ 8916.142215] Lustre: server umount lustre-OST0000 complete [ 8922.086474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8922.094328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8922.113289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8922.117369] Lustre: Skipped 1 previous similar message [ 8926.183449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8928.168490] Lustre: server umount lustre-MDT0000 complete [ 8931.297257] LustreError: 289791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8931.315576] LustreError: 289791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8932.988432] LustreError: 289772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456957 with bad export cookie 12871223836274830999 [ 8932.996019] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8933.008589] LustreError: 289772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8933.651877] Lustre: server umount lustre-MDT0001 complete [ 8946.621679] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 8950.426188] Key type lgssc unregistered [ 8950.783657] LNet: 293332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8950.791813] LNetError: 293332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8951.851409] LNet: Removed LNI 192.168.202.141@tcp [ 8952.620538] Key type .llcrypt unregistered [ 8952.623206] Key type ._llcrypt unregistered [ 8984.182563] Key type ._llcrypt registered [ 8984.184148] Key type .llcrypt registered [ 8984.241914] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 9001.109027] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9002.744157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9002.895091] alg: No test for adler32 (adler32-zlib) [ 9004.526562] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 9005.095704] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 9006.871214] Key type lgssc registered [ 9007.711885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9058.077429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9058.126180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9059.409740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9059.471629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9059.587134] Lustre: lustre-MDT0000: new disk, initializing [ 9059.640967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9059.668676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9064.212481] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9075.092867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9075.285213] Lustre: 297157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9075.418845] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9075.423135] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9075.436206] Lustre: Skipped 1 previous similar message [ 9075.584274] Lustre: lustre-MDT0001: new disk, initializing [ 9075.751077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9075.842684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9075.860952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9081.062628] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9090.507519] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9096.560392] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9102.582470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9102.798292] Lustre: lustre-OST2710: new disk, initializing [ 9102.801325] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9102.806779] Lustre: 299120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9102.919793] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9108.979790] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9110.123648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9110.143727] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9110.278052] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9118.393064] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9125.058349] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9125.194659] Lustre: lustre-OST03e8: new disk, initializing [ 9125.198648] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9125.204198] Lustre: 300187:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9125.330786] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9131.706883] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9135.248103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9135.262408] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9135.421083] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9140.317638] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9147.272276] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9147.501784] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9152.058072] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9152.333221] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9157.460186] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9157.732370] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9162.378885] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9162.568820] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9167.847853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9167.874257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9167.887702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9169.042770] Lustre: server umount lustre-MDT0000 complete [ 9171.431210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9172.850216] LustreError: 297148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457197 with bad export cookie 4058317140924824180 [ 9172.863717] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9172.865568] LustreError: 297148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9173.203039] Lustre: server umount lustre-MDT0001 complete [ 9187.280834] Lustre: server umount lustre-OST2710 complete [ 9201.587806] Lustre: server umount lustre-OST03e8 complete [ 9222.281901] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 9234.124606] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9271.220728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9271.684975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9271.745839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9271.857632] Lustre: lustre-MDT0000: new disk, initializing [ 9271.994749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9272.027251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9277.467350] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9288.383011] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9288.524838] Lustre: 305437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9288.564751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9288.568135] Lustre: Skipped 1 previous similar message [ 9288.624643] Lustre: lustre-MDT0001: new disk, initializing [ 9288.713587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9288.773143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9288.789084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9292.511913] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9302.597526] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9310.195958] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9317.818593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9318.040817] Lustre: lustre-OST0000: new disk, initializing [ 9318.043455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9318.051830] Lustre: 307407:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9318.147924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9323.926920] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9325.644651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9325.657783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9325.784785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9332.032907] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9337.441473] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9337.629248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9343.283776] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9343.548185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9346.540453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9346.541539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9346.554676] LustreError: Skipped 1 previous similar message [ 9346.558302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9346.597091] Lustre: Skipped 3 previous similar messages [ 9351.654716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9351.669928] Lustre: Skipped 1 previous similar message [ 9352.009988] Lustre: server umount lustre-OST0000 complete [ 9356.769163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9356.775468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9356.791902] Lustre: Skipped 1 previous similar message [ 9356.801563] Lustre: Skipped 2 previous similar messages [ 9362.579027] Lustre: server umount lustre-MDT0000 complete [ 9366.577077] LustreError: 305429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457390 with bad export cookie 4058317140924825860 [ 9366.583436] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9366.589601] LustreError: 305429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9367.070089] Lustre: server umount lustre-MDT0001 complete [ 9375.491241] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:43:18 (1788457398) [ 9377.035645] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9378.994964] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:43:21 (1788457401) [ 9382.924070] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9414.467456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9414.803938] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9414.879588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9415.052437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9420.180234] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9431.131465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9431.272447] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9431.327217] Lustre: 311263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9437.411791] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9459.433932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9459.578268] Lustre: Denying initial registration attempt from nid 192.168.202.141@tcp, specified as failover [ 9459.586282] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9459.593674] LustreError: 312717:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9459.598628] LustreError: 312717:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9459.603608] LustreError: 312717:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9459.717179] Lustre: server umount lustre-OST0000 complete [ 9459.719233] LustreError: 312717:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9467.363614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9467.368551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9467.405786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9467.408448] Lustre: Skipped 2 previous similar messages [ 9472.912424] Lustre: server umount lustre-MDT0000 complete [ 9476.580477] LustreError: 312425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9476.593207] LustreError: 311254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457500 with bad export cookie 4058317140924826742 [ 9476.599589] LustreError: 312425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9476.630180] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9477.078825] Lustre: server umount lustre-MDT0001 complete [ 9488.365954] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 9491.699815] Key type lgssc unregistered [ 9492.046187] LNet: 313835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9492.054274] LNetError: 313835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9492.070282] LNet: Removed LNI 192.168.202.141@tcp [ 9493.397481] Key type .llcrypt unregistered [ 9493.404318] Key type ._llcrypt unregistered [ 9508.388740] Key type ._llcrypt registered [ 9508.391135] Key type .llcrypt registered [ 9508.529930] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9509.521054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9509.693082] alg: No test for adler32 (adler32-zlib) [ 9510.815427] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 9511.151764] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 9512.920367] Key type lgssc registered [ 9513.737858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9561.854806] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9573.786210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9573.819983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9575.225332] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9575.242408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9575.371974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9580.888975] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9591.144117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9591.308851] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9591.361513] Lustre: 316599:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9591.847178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9597.982435] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9605.910982] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9612.698929] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9633.514274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9633.818552] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9633.964407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9639.757950] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9646.933963] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9650.158952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9650.171706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9650.172467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9650.190849] Lustre: Skipped 1 previous similar message [ 9655.274185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9655.283781] Lustre: Skipped 2 previous similar messages [ 9655.761119] Lustre: server umount lustre-OST0000 complete [ 9660.392497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9660.398848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9660.413463] Lustre: Skipped 1 previous similar message [ 9660.433191] Lustre: Skipped 1 previous similar message [ 9665.507130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9665.515529] Lustre: Skipped 1 previous similar message [ 9670.630435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9670.649281] Lustre: Skipped 1 previous similar message [ 9674.207139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9675.448252] Lustre: server umount lustre-MDT0000 complete [ 9679.652353] LustreError: 316593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457703 with bad export cookie 7087681664173035784 [ 9679.660728] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9679.668110] LustreError: 316593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9679.914317] Lustre: server umount lustre-MDT0001 complete [ 9692.522126] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 9697.134267] Key type lgssc unregistered [ 9697.603647] LNet: 320173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9697.609643] LNetError: 320173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9697.628587] LNet: Removed LNI 192.168.202.141@tcp [ 9699.531258] Key type .llcrypt unregistered [ 9699.536462] Key type ._llcrypt unregistered [ 9734.898774] Key type ._llcrypt registered [ 9734.903751] Key type .llcrypt registered [ 9735.111638] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9737.063647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9737.303719] alg: No test for adler32 (adler32-zlib) [ 9738.698824] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 9739.057775] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 9740.791885] Key type lgssc registered [ 9742.567356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9752.931504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9752.953520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9754.495392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9758.738801] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9763.818839] LustreError: 321593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9763.839104] LustreError: 321593:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9768.938586] LustreError: 321592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9769.509883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9770.078092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9776.330772] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9785.809438] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9792.676739] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9803.019866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9803.709194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9811.326059] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9820.651350] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9836.512208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9836.527799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9836.545621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9837.124575] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.41@tcp (stopping) [ 9839.590853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9839.593498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9841.448161] Lustre: server umount lustre-MDT0000 complete [ 9842.168867] LustreError: 321587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9844.729806] LustreError: 322520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9844.740444] LustreError: 322520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9845.035652] LustreError: 321572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457869 with bad export cookie 10221579980493135631 [ 9845.040271] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9845.046660] LustreError: 321572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9847.318340] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.41@tcp (stopping) [ 9847.331657] Lustre: Skipped 2 previous similar messages [ 9849.838243] LustreError: 322518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9849.846545] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9849.887676] LustreError: 322518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9850.023303] Lustre: Skipped 1 previous similar message [ 9851.537672] Lustre: server umount lustre-MDT0001 complete [ 9863.189057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9875.084835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9891.153808] LustreError: 323427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9891.166896] LustreError: 323427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9891.233302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9891.302052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9896.862548] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9905.639996] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9905.975596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9906.415773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9910.848143] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9911.797181] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9919.262489] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9921.683791] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9929.555142] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9931.975689] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9942.507141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9942.508429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9942.519431] LustreError: Skipped 1 previous similar message [ 9942.520192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9942.520197] Lustre: Skipped 2 previous similar messages [ 9942.562054] Lustre: Skipped 1 previous similar message [ 9944.426268] Lustre: server umount lustre-OST0000 complete [ 9952.757169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9952.759322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9952.770892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9952.785338] Lustre: Skipped 2 previous similar messages [ 9954.403736] Lustre: server umount lustre-MDT0000 complete [ 9957.869150] LustreError: 325208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9957.896202] LustreError: 325208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9958.986948] LustreError: 321573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457983 with bad export cookie 10221579980493143037 [ 9959.006976] LustreError: 321573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9959.012411] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9959.577687] Lustre: server umount lustre-MDT0001 complete [ 9971.588695] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 9975.682840] Key type lgssc unregistered [ 9976.112345] LNet: 327883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9976.124980] LNetError: 327883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9976.167838] LNet: Removed LNI 192.168.202.141@tcp [ 9977.929312] Key type .llcrypt unregistered [ 9977.932705] Key type ._llcrypt unregistered [10011.240856] Key type ._llcrypt registered [10011.243825] Key type .llcrypt registered [10011.485097] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10013.213045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10013.236885] alg: No test for adler32 (adler32-zlib) [10014.897149] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [10015.259970] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [10016.984767] Key type lgssc registered [10018.892945] Lustre: Echo OBD driver; http://www.lustre.org/ [10028.349443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10028.373912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10029.863781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10034.795355] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10040.291720] LustreError: 329295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10040.318082] LustreError: 329295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10044.873047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10045.232247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10049.913143] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10057.790900] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10063.734672] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10067.260791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10067.284951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10067.298602] Lustre: Skipped 1 previous similar message [10070.371040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10070.376513] Lustre: Skipped 1 previous similar message [10073.694538] Lustre: server umount lustre-MDT0000 complete [10075.618900] LustreError: 329292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10075.644052] LustreError: 329292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10077.598760] LustreError: 329277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788458101 with bad export cookie 12560440343694568046 [10077.607072] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10077.619636] LustreError: 329277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10078.056500] Lustre: server umount lustre-MDT0001 complete [10095.487524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10095.815789] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10095.835697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10096.000562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10100.670903] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10110.673662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10110.768209] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10110.787625] Lustre: 331884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10111.131885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10116.697580] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10124.991778] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10132.018532] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10142.779266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10142.957346] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10142.978938] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10143.001666] LustreError: 333768:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [10143.023595] LustreError: 333768:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10143.030919] LustreError: 333768:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10143.083267] Lustre: server umount lustre-OST0000 complete [10143.086836] LustreError: 333768:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10155.896460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10156.256936] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10156.615909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10163.313679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10165.089407] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10173.308918] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10180.847620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10181.006985] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10181.120946] Lustre: lustre-OST0001: new disk, initializing [10181.125049] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10181.134047] Lustre: 335309:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10181.210928] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10189.303448] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10190.509496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10190.522280] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10190.714538] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10198.192402] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10206.184433] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10206.192094] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10206.193029] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10206.196103] LustreError: Skipped 1 previous similar message [10206.212235] Lustre: Skipped 2 previous similar messages [10207.540775] Lustre: server umount lustre-OST0001 complete [10211.301473] LustreError: 334235:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10211.318871] LustreError: 334235:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10214.368273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10214.375229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10214.393949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10214.396296] Lustre: Skipped 1 previous similar message [10216.420358] LustreError: 334236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10216.454927] LustreError: 334236:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10221.556103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10221.558652] Lustre: Skipped 4 previous similar messages [10221.561534] LustreError: 334235:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10221.570275] LustreError: 334235:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10226.143153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10226.398355] Lustre: server umount lustre-OST0000 complete [10231.790143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10231.811493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10231.814439] Lustre: Skipped 2 previous similar messages [10231.827100] Lustre: Skipped 1 previous similar message [10236.797252] Lustre: server umount lustre-MDT0000 complete [10236.898719] LustreError: 331890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10236.926831] LustreError: 331890:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10242.918322] LustreError: 331876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788458267 with bad export cookie 12560440343694568599 [10242.938926] LustreError: 331876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10242.950383] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10243.586615] Lustre: server umount lustre-MDT0001 complete [10258.945629] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [10262.341733] Key type lgssc unregistered [10262.651866] LNet: 337272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10262.664378] LNetError: 337272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10262.694418] LNet: Removed LNI 192.168.202.141@tcp [10263.691325] Key type .llcrypt unregistered [10263.694113] Key type ._llcrypt unregistered [10344.985797] Key type ._llcrypt registered [10344.989829] Key type .llcrypt registered [10345.109204] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [10363.608968] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10365.636284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10365.679629] alg: No test for adler32 (adler32-zlib) [10366.814821] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [10367.129983] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [10368.927307] Key type lgssc registered [10370.569761] Lustre: Echo OBD driver; http://www.lustre.org/ [10407.365538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10407.410826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10408.682363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10408.730883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10408.839592] Lustre: lustre-MDT0000: new disk, initializing [10408.980204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10409.032843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10414.552198] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10426.092132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10426.251975] Lustre: 342201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10426.354023] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10426.375959] Lustre: Skipped 1 previous similar message [10426.561918] Lustre: lustre-MDT0001: new disk, initializing [10426.666579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10426.718891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10426.727470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10431.753958] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10443.448942] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10451.270745] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10460.338832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10460.738702] Lustre: lustre-OST0000: new disk, initializing [10460.745928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10460.765402] Lustre: 344184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10460.935660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10467.926046] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10470.564330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10470.573280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10470.764660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10478.697631] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10484.809848] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10485.054608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10491.700350] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10492.165701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10496.480371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10496.481465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10496.483396] LustreError: Skipped 1 previous similar message [10496.485928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10496.519804] Lustre: Skipped 1 previous similar message [10499.558810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10499.575848] Lustre: Skipped 2 previous similar messages [10500.800658] Lustre: server umount lustre-OST0000 complete [10508.770660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10508.782099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10508.799666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10509.794886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10511.405170] Lustre: server umount lustre-MDT0000 complete [10515.664334] LustreError: 342196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788458539 with bad export cookie 4079026852847420458 [10515.685787] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10516.032212] Lustre: server umount lustre-MDT0001 complete [10525.435309] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 14:02:27 (1788458547) [10533.796867] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 14:02:36 (1788458556) [10543.450489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10543.997426] LustreError: 346946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10544.027144] LustreError: 346946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10544.184492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10549.222765] LustreError: 346947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10549.244288] LustreError: 346947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10550.058321] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10554.336334] LustreError: 346946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10559.457880] LustreError: 346947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10560.241303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10560.765520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10566.304759] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10574.212915] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10580.017524] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10588.593610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10589.290072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10596.942459] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10606.585646] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10615.573494] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10617.986259] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10619.672551] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10621.648481] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10623.810539] Lustre: Failing over lustre-MDT0000 [10624.140194] Lustre: server umount lustre-MDT0000 complete [10624.553702] LustreError: 346941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10626.530719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10634.742154] LustreError: 346941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10634.762218] LustreError: 346941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10641.890340] Lustre: 340424:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788458650/real 1788458650] req@0000000077beabd9 x1875334545878912/t0(0) o400->MGC192.168.202.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788458666 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10641.936651] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10644.771483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10651.128154] LustreError: 346943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10651.164855] LustreError: 346943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [10652.603861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10652.700138] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10656.254391] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10657.807266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10657.857371] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10657.953884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10659.600783] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10674.177094] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10676.442523] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10680.384973] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10682.543594] Lustre: Failing over lustre-MDT0000 [10683.362738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10683.366625] LustreError: 349074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10683.377846] Lustre: Skipped 3 previous similar messages [10683.419135] LustreError: 349074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10684.813802] Lustre: server umount lustre-MDT0000 complete [10699.743126] Lustre: 340426:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788458707/real 1788458707] req@000000005dce80bd x1875334545900416/t0(0) o400->MGC192.168.202.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788458723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10699.777433] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10704.171163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10709.984803] LustreError: 340423:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001bee2b39 x1875334545905792/t0(0) o250->MGC192.168.202.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10710.349246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10710.449465] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10713.588412] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10715.624625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10715.629566] Lustre: Skipped 2 previous similar messages [10715.664922] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10715.721881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10716.047821] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10725.441854] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10726.766711] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10736.097171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10736.101409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10736.108703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10736.108711] Lustre: Skipped 2 previous similar messages [10736.142466] Lustre: Skipped 3 previous similar messages [10738.713197] Lustre: server umount lustre-MDT0000 complete [10743.223863] LustreError: 346927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788458767 with bad export cookie 4079026852847423167 [10743.232711] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10743.234220] LustreError: 346927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10743.661560] Lustre: server umount lustre-MDT0001 complete [10758.676220] Lustre: server umount lustre-OST0000 complete [10774.579249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10775.079219] LustreError: 353291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10775.099402] LustreError: 353291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [10775.172261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10780.526935] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10789.912665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10796.023356] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10803.742454] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10808.809602] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10815.585816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10820.030851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10823.916966] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10832.089553] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10844.141481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [10844.146775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10844.154249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10844.158494] Lustre: Skipped 2 previous similar messages [10848.284785] Lustre: server umount lustre-OST0000 complete [10855.917446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10855.927891] Lustre: Skipped 3 previous similar messages [10859.197375] Lustre: server umount lustre-MDT0000 complete [10863.733439] LustreError: 354026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788458887 with bad export cookie 4079026852847423916 [10863.742173] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10863.751638] LustreError: 354026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10864.268819] Lustre: server umount lustre-MDT0001 complete [10875.342581] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [10878.550202] Key type lgssc unregistered [10878.843767] LNet: 356824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10878.849752] LNetError: 356824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10878.862947] LNet: Removed LNI 192.168.202.141@tcp [10879.849050] Key type .llcrypt unregistered [10879.855025] Key type ._llcrypt unregistered [10908.649792] Key type ._llcrypt registered [10908.654427] Key type .llcrypt registered [10908.790473] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10910.465713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10910.563422] alg: No test for adler32 (adler32-zlib) [10911.533587] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [10911.763647] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [10913.418464] Key type lgssc registered [10915.222743] Lustre: Echo OBD driver; http://www.lustre.org/ [10925.122063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10925.145440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10926.702268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10931.962574] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10936.303967] LustreError: 358337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10936.322942] LustreError: 358337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10941.372439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10941.413314] LustreError: 358338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10941.742574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10946.720956] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10954.126213] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10960.078341] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10967.186886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10967.463221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10971.586135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10973.832398] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10981.912307] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10997.742598] Lustre: server umount lustre-MDT0000 complete [10998.248022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10998.258324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10998.288193] LustreError: 358338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11001.317089] LustreError: 358337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11001.320503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11001.361087] Lustre: Skipped 1 previous similar message [11001.817886] LustreError: 358317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788459026 with bad export cookie 14725716955732414095 [11001.826638] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11001.837976] LustreError: 358317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11002.182672] Lustre: server umount lustre-MDT0001 complete [11016.850422] Lustre: server umount lustre-OST0000 complete [11030.584942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11031.102829] LustreError: 362070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11031.121338] LustreError: 362070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11031.203193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11036.140691] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11041.765827] LustreError: 362071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11041.794956] LustreError: 362071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11045.306095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11045.595110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11049.881050] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11057.913589] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11063.838675] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11070.349561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11070.751215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11075.928868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11077.854598] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11084.672449] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11097.057686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11097.068229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11097.082110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11100.643600] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11100.650264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11100.674271] Lustre: Skipped 2 previous similar messages [11103.399606] Lustre: server umount lustre-MDT0000 complete [11105.762514] LustreError: 363118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11105.781970] LustreError: 363118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11107.533359] LustreError: 362805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788459131 with bad export cookie 14725716955732415215 [11107.540699] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11107.556896] LustreError: 362805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11108.052504] Lustre: server umount lustre-MDT0001 complete [11122.920807] Lustre: server umount lustre-OST0000 complete [11138.780986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11139.347956] LustreError: 365982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11139.455691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11144.275979] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11152.358821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11158.188724] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11165.835362] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11171.659682] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11177.948649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11178.221727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11178.224385] Lustre: Skipped 1 previous similar message [11185.305582] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11185.394177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11193.659532] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11200.993522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11201.011786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11201.025353] Lustre: Skipped 1 previous similar message [11201.030043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11203.046778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11203.050120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11205.359483] Lustre: server umount lustre-OST0000 complete [11209.184718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11209.191593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11209.199573] Lustre: Skipped 1 previous similar message [11209.204372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11209.211493] Lustre: Skipped 1 previous similar message [11215.567265] Lustre: server umount lustre-MDT0000 complete [11218.404663] LustreError: 365983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11218.438802] LustreError: 365983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11219.341678] LustreError: 366718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788459243 with bad export cookie 14725716955732416237 [11219.345312] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11219.355042] LustreError: 366718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11219.747538] Lustre: server umount lustre-MDT0001 complete [11232.595115] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [11235.563387] Key type lgssc unregistered [11235.879558] LNet: 369515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11235.888095] LNetError: 369515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11236.967372] LNet: Removed LNI 192.168.202.141@tcp [11238.158616] Key type .llcrypt unregistered [11238.162365] Key type ._llcrypt unregistered [11267.640909] Key type ._llcrypt registered [11267.643753] Key type .llcrypt registered [11267.738668] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [11269.474146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11269.575546] alg: No test for adler32 (adler32-zlib) [11270.688951] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [11270.943492] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [11272.623966] Key type lgssc registered [11274.086917] Lustre: Echo OBD driver; http://www.lustre.org/ [11283.006626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11283.019401] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11283.025414] LustreError: 370954:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11283.044843] LustreError: 370954:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11283.047734] LustreError: 370954:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11283.054918] LustreError: 370954:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11283.062364] LustreError: 370954:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11291.362647] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11291.373157] LustreError: 371351:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11291.395159] LustreError: 371351:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11291.404622] LustreError: 371351:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11291.413665] LustreError: 371351:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11291.423840] LustreError: 371351:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11312.356650] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [11315.128416] Key type lgssc unregistered [11315.435521] LNet: 372354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11315.443117] LNetError: 372354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11315.457091] LNet: Removed LNI 192.168.202.141@tcp [11316.579140] Key type .llcrypt unregistered [11316.581976] Key type ._llcrypt unregistered [11337.715805] Key type ._llcrypt registered [11337.719393] Key type .llcrypt registered [11337.918467] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [11355.177492] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [11357.222711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11357.251631] alg: No test for adler32 (adler32-zlib) [11358.388562] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [11358.659183] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [11360.335566] Key type lgssc registered [11361.642359] Lustre: Echo OBD driver; http://www.lustre.org/ [11401.415943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11401.458531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11402.925315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11403.011909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11403.234724] Lustre: lustre-MDT0000: new disk, initializing [11403.381433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11403.424356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11408.225883] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11419.130049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11419.320025] Lustre: 375504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11419.432137] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11419.439700] Lustre: Skipped 1 previous similar message [11419.577931] Lustre: lustre-MDT0001: new disk, initializing [11419.793752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11419.864376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11419.887106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11425.712313] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11436.910389] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11442.887956] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11450.892181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11451.331700] Lustre: lustre-OST0000: new disk, initializing [11451.337389] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11451.346538] Lustre: 377474:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11451.513634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11458.752623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11458.761356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11458.942453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11459.462327] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11469.218734] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11474.335502] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11474.621306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11480.116926] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11480.379052] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11484.650404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11484.652757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11484.668026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11484.691068] Lustre: Skipped 1 previous similar message [11488.851507] Lustre: server umount lustre-OST0000 complete [11494.889417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11494.910469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11494.913317] Lustre: Skipped 1 previous similar message [11500.001430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11500.006787] Lustre: Skipped 1 previous similar message [11504.107872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11504.114926] Lustre: Skipped 1 previous similar message [11507.169240] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11507.563985] Lustre: server umount lustre-MDT0000 complete [11511.532928] LustreError: 376301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788459535 with bad export cookie 14276030444544157951 [11511.543539] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11511.546591] LustreError: 376301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11512.000624] Lustre: server umount lustre-MDT0001 complete [11521.577806] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11523.706066] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 14:19:06 (1788459546) [11530.655305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11531.085615] LustreError: 379937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11531.106538] LustreError: 379937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11531.217737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11536.122332] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11536.354475] LustreError: 379938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11541.476538] LustreError: 379937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11543.718833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11544.162930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11549.050332] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11557.515557] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11563.986470] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11572.078436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11572.708823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11578.753166] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11588.187333] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11596.359644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11596.554253] Lustre: lustre-OST0001: new disk, initializing [11596.557849] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11596.570869] Lustre: 382774:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11596.674798] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11602.587188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11602.597357] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11602.807167] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11604.907255] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11616.019047] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11623.392779] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11623.396615] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11623.410837] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11623.432887] Lustre: Skipped 2 previous similar messages [11627.206668] Lustre: server umount lustre-OST0001 complete [11628.527198] LustreError: 381757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11633.636582] LustreError: 381756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11633.673871] LustreError: 381756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11643.890602] LustreError: 381757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11643.924218] LustreError: 381757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11645.409393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11645.413148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11645.444650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11645.455581] Lustre: Skipped 2 previous similar messages [11648.483423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11651.042434] Lustre: server umount lustre-OST0000 complete [11656.672403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11656.682558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11656.696278] Lustre: Skipped 1 previous similar message [11664.247027] Lustre: server umount lustre-MDT0000 complete [11664.362113] LustreError: 381779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11664.381422] LustreError: 381779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11668.475398] LustreError: 379919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788459692 with bad export cookie 14276030444544158826 [11668.486569] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11668.487795] LustreError: 379919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11675.152558] Lustre: server umount lustre-MDT0001 complete [11690.258584] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [11693.974421] Key type lgssc unregistered [11694.369988] LNet: 384759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11694.375913] LNetError: 384759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11694.391975] LNet: Removed LNI 192.168.202.141@tcp [11695.445498] Key type .llcrypt unregistered [11695.449122] Key type ._llcrypt unregistered [11750.900151] Key type ._llcrypt registered [11750.904341] Key type .llcrypt registered [11751.003260] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [11752.449036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11752.480033] alg: No test for adler32 (adler32-zlib) [11753.623305] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [11753.931777] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [11755.696787] Key type lgssc registered [11757.143710] Lustre: Echo OBD driver; http://www.lustre.org/ [11764.837230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11764.851534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11766.121782] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11766.149861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11766.431726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11771.845208] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11780.925165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11781.091158] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11781.144142] Lustre: 387558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11781.506389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11785.994967] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11794.783625] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11801.826570] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11803.616582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11803.623315] Lustre: Skipped 1 previous similar message [11803.625884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11803.627904] Lustre: Skipped 1 previous similar message [11808.741249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11808.746639] Lustre: Skipped 1 previous similar message [11809.855550] Lustre: server umount lustre-MDT0000 complete [11813.172826] LustreError: 388300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788459837 with bad export cookie 11372252055833737706 [11813.175821] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11813.189698] LustreError: 388300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11813.607847] Lustre: server umount lustre-MDT0001 complete [11824.281796] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 14:24:07 (1788459847) [11831.314972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11832.079034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11837.419244] LustreError: 390070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11837.459925] LustreError: 390070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11837.827858] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11842.531225] LustreError: 390069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11847.051378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11847.349721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11851.565717] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11859.733201] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11866.460764] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11874.854531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11875.225155] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11875.673315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11883.984622] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11890.894316] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11900.156844] Lustre: Permanently deactivating lustre-OST0000 [11900.192346] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11906.162920] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11910.002594] LustreError: 393011:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11910.026928] LustreError: 393011:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11910.035956] LustreError: 393011:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11912.793042] LustreError: 393060:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11912.806407] LustreError: 393060:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11917.734966] Lustre: server umount lustre-OST0000 complete [11920.739292] LustreError: 393309:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11920.743753] LustreError: 393309:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11924.449815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11924.455103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11924.483036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11925.487201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11930.600160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11930.602751] Lustre: Skipped 2 previous similar messages [11932.861945] Lustre: server umount lustre-MDT0000 complete [11935.718678] LustreError: 390064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11935.744044] LustreError: 390064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11936.260355] LustreError: 390049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788459960 with bad export cookie 11372252055833738280 [11936.270052] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11936.277204] LustreError: 390049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11946.728923] Lustre: server umount lustre-MDT0001 complete [11956.261801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11961.251969] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11964.544545] LustreError: 394469:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11972.029692] Lustre: 394809:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.141@tcp [11974.241457] Lustre: server umount MGS complete [11987.963037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11988.654587] LustreError: 395533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11988.703742] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11988.714628] Lustre: Skipped 1 previous similar message [11988.785605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11994.081312] LustreError: 395534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11994.363546] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12004.326217] LustreError: 395533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12004.358323] LustreError: 395533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12004.570397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12004.941767] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12005.083299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12010.744438] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12019.095796] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12026.296911] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12028.108982] Lustre: Permanently reactivating lustre-OST0000 [12028.125572] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12028.130732] Lustre: Skipped 2 previous similar messages [12057.055117] LustreError: 397314:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12066.067521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12066.550171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12073.723417] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12080.803783] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12087.264773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12087.269741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12087.280590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12087.284489] Lustre: Skipped 1 previous similar message [12092.722069] Lustre: server umount lustre-OST0000 complete [12096.995967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12097.017374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12097.025444] Lustre: Skipped 1 previous similar message [12097.030156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12097.036143] Lustre: Skipped 2 previous similar messages [12102.838262] Lustre: server umount lustre-MDT0000 complete [12103.649289] LustreError: 396491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12103.662384] LustreError: 396491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12106.811184] LustreError: 395513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788460131 with bad export cookie 11372252055833739337 [12106.815024] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12106.824556] LustreError: 395513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12107.176996] Lustre: server umount lustre-MDT0001 complete [12119.643329] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [12122.973950] Key type lgssc unregistered [12123.254574] LNet: 399897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12123.260794] LNetError: 399897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12123.275071] LNet: Removed LNI 192.168.202.141@tcp [12124.343534] Key type .llcrypt unregistered [12124.346230] Key type ._llcrypt unregistered [12147.714606] Key type ._llcrypt registered [12147.716041] Key type .llcrypt registered [12147.832876] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [12165.939510] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [12167.436684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12167.489509] alg: No test for adler32 (adler32-zlib) [12168.619681] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [12169.006881] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [12170.775235] Key type lgssc registered [12172.215837] Lustre: Echo OBD driver; http://www.lustre.org/ [12214.513810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12214.569033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12215.857870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12215.905269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12216.021137] Lustre: lustre-MDT0000: new disk, initializing [12216.130620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12216.148856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12221.795672] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12232.555442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12232.650028] Lustre: 403056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12232.695237] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12232.699803] Lustre: Skipped 1 previous similar message [12232.730424] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12232.780676] Lustre: lustre-MDT0001: new disk, initializing [12232.881259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12232.913637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12232.922588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12238.451834] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12250.866890] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12258.347913] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12267.103417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12267.628394] Lustre: lustre-OST0000: new disk, initializing [12267.636639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12267.649779] Lustre: 405027:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12267.809259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12276.383273] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12276.833710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12276.839744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12276.995185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12288.945268] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12295.167808] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12295.522847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12301.747570] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12302.051960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12307.947728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12307.949881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12307.954092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12307.991319] Lustre: Skipped 1 previous similar message [12308.022377] Lustre: Skipped 1 previous similar message [12310.649808] Lustre: server umount lustre-OST0000 complete [12315.115086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12315.119147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12315.136115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12318.176735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12318.179254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12321.369833] Lustre: server umount lustre-MDT0000 complete [12325.595448] LustreError: 405024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788460349 with bad export cookie 13618523062077196044 [12325.612135] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12325.614251] LustreError: 405024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12325.988558] Lustre: server umount lustre-MDT0001 complete [12335.175567] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12337.137810] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 14:32:39 (1788460359) [12345.004512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12345.453268] LustreError: 407489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12345.471132] LustreError: 407489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12345.533319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12350.960453] LustreError: 407490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12351.044837] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12355.046895] LustreError: 407489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12360.167980] LustreError: 407490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12360.738286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12361.191790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12366.472847] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12374.644262] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12381.113927] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12388.603814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12388.995984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12396.748625] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12404.699494] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12406.892175] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12410.594946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12416.992843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12416.994992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12417.002232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12417.002240] Lustre: Skipped 1 previous similar message [12417.063770] Lustre: Skipped 1 previous similar message [12422.603958] Lustre: server umount lustre-OST0000 complete [12427.744059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12427.759589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12427.797964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12427.800760] Lustre: Skipped 3 previous similar messages [12442.079323] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12442.600070] Lustre: server umount lustre-MDT0000 complete [12442.620778] LustreError: 408457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12442.646813] LustreError: 408457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12447.314415] LustreError: 407470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788460471 with bad export cookie 13618523062077196926 [12447.316357] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12447.332866] LustreError: 407470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12448.107378] Lustre: server umount lustre-MDT0001 complete [12461.964783] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [12465.678386] Key type lgssc unregistered [12466.079661] LNet: 411216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12466.092362] LNetError: 411216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12466.115760] LNet: Removed LNI 192.168.202.141@tcp [12467.108246] Key type .llcrypt unregistered [12467.109858] Key type ._llcrypt unregistered [12494.289256] Key type ._llcrypt registered [12494.295342] Key type .llcrypt registered [12494.502916] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [12494.854725] Key type .llcrypt unregistered [12494.863155] Key type ._llcrypt unregistered [12516.803649] Key type ._llcrypt registered [12516.806361] Key type .llcrypt registered [12517.013981] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [12518.577338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12518.719474] alg: No test for adler32 (adler32-zlib) [12520.269901] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [12520.663798] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [12522.432068] Key type lgssc registered [12524.073303] Lustre: Echo OBD driver; http://www.lustre.org/ [12533.096271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12533.127050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12534.623909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12538.674400] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12543.974243] LustreError: 413437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12544.006393] LustreError: 413437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12547.672558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12548.163606] LustreError: 413438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12548.198549] LustreError: 413438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12548.215845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12553.185476] LustreError: 413437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12553.225162] LustreError: 413437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12556.170194] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12558.322114] LustreError: 413438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12558.333479] LustreError: 413438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12559.349815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12564.454056] LustreError: 413438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12564.485720] LustreError: 413438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12565.115710] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12573.458360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12574.102874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12579.937702] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12594.663899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12594.672886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12594.688823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12594.712737] Lustre: Skipped 1 previous similar message [12596.762183] Lustre: server umount lustre-OST0000 complete [12604.275950] Lustre: server umount lustre-MDT0000 complete [12604.899796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12604.902092] LustreError: 413434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12604.925834] Lustre: Skipped 1 previous similar message [12604.975988] LustreError: 413434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12608.388508] LustreError: 415831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788460632 with bad export cookie 6013278128096144452 [12608.398792] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12608.403621] LustreError: 415831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12609.041343] Lustre: server umount lustre-MDT0001 complete [12621.032664] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [12624.400387] Key type lgssc unregistered [12624.827836] LNet: 416582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12624.832396] LNetError: 416582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12624.845897] LNet: Removed LNI 192.168.202.141@tcp [12626.190990] Key type .llcrypt unregistered [12626.193566] Key type ._llcrypt unregistered [12656.233658] Key type ._llcrypt registered [12656.235161] Key type .llcrypt registered [12656.380807] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [12657.885419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12657.967817] alg: No test for adler32 (adler32-zlib) [12659.165490] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [12659.470552] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [12661.246114] Key type lgssc registered [12662.814107] Lustre: Echo OBD driver; http://www.lustre.org/ [12673.494588] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12673.520636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12689.183406] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12694.303521] LustreError: 417973:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.141@tcp: failed processing log, type 4: rc = -110 [12721.055349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12728.635652] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12736.635694] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12744.451299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12745.284968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12745.414115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12750.312382] LustreError: 417998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12750.333244] LustreError: 417998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12750.582304] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12755.430757] LustreError: 417999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12756.480759] LustreError: 419009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12756.513414] LustreError: 419009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12760.963920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12761.494684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12767.740330] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12777.264041] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12783.755513] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12796.385595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12796.391559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12796.401973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12797.411564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12797.435878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12797.446593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12798.655134] Lustre: server umount lustre-OST0000 complete [12806.212945] Lustre: server umount lustre-MDT0000 complete [12807.653651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12807.664817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12807.676654] LustreError: 419011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12807.702879] LustreError: 419011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12810.709619] LustreError: 419751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788460834 with bad export cookie 11985668337269395479 [12810.713981] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12810.721481] LustreError: 419751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12811.258971] Lustre: server umount lustre-MDT0001 complete [12826.411837] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [12830.491379] Key type lgssc unregistered [12830.951548] LNet: 421543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12830.962194] LNetError: 421543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12832.044710] LNet: Removed LNI 192.168.202.141@tcp [12833.242456] Key type .llcrypt unregistered [12833.244610] Key type ._llcrypt unregistered [12864.505863] Key type ._llcrypt registered [12864.509738] Key type .llcrypt registered [12864.728711] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [12866.386586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12866.489967] alg: No test for adler32 (adler32-zlib) [12867.563205] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [12867.923511] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [12869.831351] Key type lgssc registered [12871.541234] Lustre: Echo OBD driver; http://www.lustre.org/ [12882.566798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12882.620808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12884.102033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12890.443890] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12894.689884] LustreError: 422957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12894.717872] LustreError: 422957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12899.820623] LustreError: 422956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12901.747683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12902.429779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12908.399260] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12918.530784] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12924.957809] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12932.761214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12933.072457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12941.434329] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12951.133939] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12954.633638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12963.348336] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.41@tcp (stopping) [12963.809341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12963.833145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12963.872971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12964.865246] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12964.891556] Lustre: Skipped 1 previous similar message [12964.918668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12964.941185] Lustre: Skipped 2 previous similar messages [12967.060726] Lustre: server umount lustre-MDT0000 complete [12968.438616] LustreError: 422953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12970.004356] LustreError: 422938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.41@tcp arrived at 1788460994 with bad export cookie 12338716282604261561 [12972.558976] LustreError: 422936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788460996 with bad export cookie 12338716282604260896 [12972.574163] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12972.577237] LustreError: 422936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12973.036555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12973.050290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12973.070873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12975.073500] LustreError: 422956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12975.092842] LustreError: 422956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12977.121057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12977.123823] Lustre: Skipped 1 previous similar message [12978.856111] Lustre: server umount lustre-OST0000 complete [12982.247059] LustreError: 422956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12982.266940] LustreError: 422956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12987.100773] Lustre: server umount lustre-MDT0001 complete [12999.259996] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13002.510381] Key type lgssc unregistered [13002.949286] LNet: 426644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13002.958120] LNetError: 426644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13002.984283] LNet: Removed LNI 192.168.202.141@tcp [13004.086695] Key type .llcrypt unregistered [13004.089154] Key type ._llcrypt unregistered [13032.545325] Key type ._llcrypt registered [13032.547613] Key type .llcrypt registered [13032.666310] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [13034.200113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13034.347883] alg: No test for adler32 (adler32-zlib) [13035.454711] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [13035.780660] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [13037.512726] Key type lgssc registered [13038.844588] Lustre: Echo OBD driver; http://www.lustre.org/ [13049.013136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13049.054871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13050.840295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13057.325559] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13061.088428] LustreError: 428059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13061.114373] LustreError: 428059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13066.223856] LustreError: 428060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13067.019788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13067.535429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13072.069741] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13080.204246] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13085.066491] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13093.726895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13093.951338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13096.002569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13100.625961] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13110.080486] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13117.428340] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13117.448583] Lustre: Skipped 1 previous similar message [13117.449234] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13117.464391] Lustre: Skipped 1 previous similar message [13117.970794] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.41@tcp (stopping) [13122.066814] Lustre: server umount lustre-MDT0001 complete [13122.529446] LustreError: 428787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13122.577698] LustreError: 428787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13127.650942] LustreError: 428060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13127.692116] LustreError: 428060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13131.745082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13131.757273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13131.796808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13132.776558] LustreError: 428059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13135.803334] Lustre: server umount lustre-OST0000 complete [13140.019707] Lustre: server umount lustre-MDT0000 complete [13155.752699] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13159.199342] Key type lgssc unregistered [13159.506713] LNet: 431745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13159.519628] LNetError: 431745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13159.551547] LNet: Removed LNI 192.168.202.141@tcp [13160.999228] Key type .llcrypt unregistered [13161.002170] Key type ._llcrypt unregistered [13189.613272] Key type ._llcrypt registered [13189.615938] Key type .llcrypt registered [13189.725044] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [13207.375527] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [13208.266835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13208.406429] alg: No test for adler32 (adler32-zlib) [13209.774195] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [13210.098858] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [13211.839176] Key type lgssc registered [13213.285728] Lustre: Echo OBD driver; http://www.lustre.org/ [13258.697439] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [13275.533879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13275.594209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13276.964473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13277.000426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13277.104527] Lustre: lustre-MDT0000: new disk, initializing [13277.206201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13277.224377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13281.751774] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13299.283798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13299.571863] Lustre: lustre-OST0000: new disk, initializing [13299.577758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13299.589309] Lustre: Skipped 1 previous similar message [13299.601410] Lustre: 436775:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13299.705201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13306.645407] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13309.045289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13309.063288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13309.092382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13316.389849] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13334.633924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13334.743898] Lustre: 435758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13334.802893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13334.861492] Lustre: lustre-MDT0001: new disk, initializing [13334.965978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13334.995740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13335.012100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13339.625293] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13351.141817] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13351.448543] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13357.533546] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13357.806410] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13364.193308] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13364.200442] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13364.218284] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13364.234290] Lustre: Skipped 1 previous similar message [13364.255073] Lustre: Skipped 1 previous similar message [13365.279450] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.41@tcp (stopping) [13369.315344] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13369.324772] Lustre: Skipped 1 previous similar message [13370.209423] Lustre: server umount lustre-MDT0001 complete [13375.486639] LustreError: 435764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13375.496403] LustreError: 435764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13379.554388] LustreError: 437664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13379.569817] LustreError: 437664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13384.674041] LustreError: 435764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13384.693813] LustreError: 435764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13389.800868] LustreError: 435764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13389.819462] LustreError: 435764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13390.818135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13390.829749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13390.838780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13394.914525] LustreError: 435768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13394.920788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13396.667670] Lustre: server umount lustre-OST0000 complete [13408.129232] Lustre: server umount lustre-MDT0000 complete [13427.119308] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13430.590323] Key type lgssc unregistered [13431.107753] LNet: 440525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13431.117028] LNetError: 440525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13431.151715] LNet: Removed LNI 192.168.202.141@tcp [13432.630893] Key type .llcrypt unregistered [13432.632764] Key type ._llcrypt unregistered [13483.688303] Key type ._llcrypt registered [13483.694213] Key type .llcrypt registered [13483.896442] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13484.478462] Key type .llcrypt unregistered [13484.480912] Key type ._llcrypt unregistered [13504.555986] Key type ._llcrypt registered [13504.558401] Key type .llcrypt registered [13504.636192] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [13506.131710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13506.183536] alg: No test for adler32 (adler32-zlib) [13507.432987] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [13507.837673] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [13509.615403] Key type lgssc registered [13511.245888] Lustre: Echo OBD driver; http://www.lustre.org/ [13541.424450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13541.461795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13542.661308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13542.682851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13542.736254] Lustre: lustre-MDT0000: new disk, initializing [13542.785520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13542.799795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13547.804928] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13559.074734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13559.263358] Lustre: 443894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13559.419471] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13559.428903] Lustre: Skipped 1 previous similar message [13559.506462] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13559.550672] Lustre: lustre-MDT0001: new disk, initializing [13559.689127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13559.748982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13559.761715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13565.016543] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13575.265721] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13582.050581] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13589.869959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13590.204397] Lustre: lustre-OST0000: new disk, initializing [13590.207477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13590.214777] Lustre: 445867:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13590.329566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13597.802750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13597.816735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13597.968334] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13597.993602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13609.282680] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13618.662172] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13618.688881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13618.689563] Lustre: Skipped 2 previous similar messages [13618.703434] Lustre: Skipped 2 previous similar messages [13624.010107] Lustre: server umount lustre-MDT0000 complete [13627.909484] LustreError: 443888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788461652 with bad export cookie 2804515780829183401 [13627.910391] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13627.955034] LustreError: 443888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13628.287186] Lustre: server umount lustre-MDT0001 complete [13643.709981] Lustre: server umount lustre-OST0000 complete [13654.344971] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:54:37 (1788461677) [13664.423402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13665.044201] LustreError: 447896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13665.059256] LustreError: 447896:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13665.104594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13670.364222] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13670.373112] LustreError: 447897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13675.489607] LustreError: 447896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13680.618633] LustreError: 447897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13681.732943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13682.336228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13688.729531] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13697.225865] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13703.638503] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13723.053665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13723.333091] Lustre: Found index 0 for lustre-OST0000, updating log [13723.673106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13730.805409] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13736.075093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13741.461506] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13748.194667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13748.205348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13748.244518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13749.297569] Lustre: server umount lustre-OST0000 complete [13769.188525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13769.203494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13769.216601] Lustre: Skipped 1 previous similar message [13769.221574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13769.229826] Lustre: Skipped 1 previous similar message [13772.758696] Lustre: server umount lustre-MDT0000 complete [13775.849845] LustreError: 447896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13775.874722] LustreError: 447896:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13777.174609] LustreError: 449740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788461801 with bad export cookie 2804515780829184794 [13777.184114] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13777.200795] LustreError: 449740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13777.709399] Lustre: server umount lustre-MDT0001 complete [13784.351594] LNet: 1 local NIs in recovery (showing 1): 192.168.202.141@tcp [13788.088495] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:56:50 (1788461810) [14418.026747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14418.538727] LustreError: 454323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14418.632640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14423.805938] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [14424.056492] LustreError: 454324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14428.130388] LustreError: 454323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14433.942931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14434.275628] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [14434.296987] Lustre: Skipped 2 previous similar messages [14434.534769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14439.566428] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [14447.955625] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14454.045172] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14474.427550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14474.541823] LNet: 1 local NIs in recovery (showing 1): 192.168.202.141@tcp [14474.673229] Lustre: Found index 0 for lustre-OST0000, updating log [14474.980694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14483.018578] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [14491.562323] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14494.630744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14494.655868] Lustre: Skipped 1 previous similar message [14494.671068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14499.997609] Lustre: server umount lustre-OST0000 complete [14505.376546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14505.379774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14505.387729] Lustre: Skipped 1 previous similar message [14505.399026] Lustre: Skipped 1 previous similar message [14507.493282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14507.496966] Lustre: Skipped 2 previous similar messages [14511.008457] Lustre: server umount lustre-MDT0000 complete [14511.584805] LustreError: 455064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14511.611495] LustreError: 455064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14515.209771] LustreError: 454305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788462539 with bad export cookie 2804515780829185879 [14515.233340] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14515.241710] LustreError: 454305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14515.950668] Lustre: server umount lustre-MDT0001 complete [14533.924343] LNet: 1 local NIs in recovery (showing 1): 192.168.202.141@tcp [14546.711809] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [14557.227518] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [14598.082246] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 15:10:20 (1788462620) [14618.889073] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [14622.219672] Key type lgssc unregistered [14622.644779] LNet: 460936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14622.656257] LNetError: 460936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14734.567169] LNet: Removed LNI 192.168.202.141@tcp [14735.857437] Key type .llcrypt unregistered [14735.860376] Key type ._llcrypt unregistered [14755.677734] Key type ._llcrypt registered [14755.681197] Key type .llcrypt registered [14755.819158] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [14757.072433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14757.309500] alg: No test for adler32 (adler32-zlib) [14758.426664] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [14758.784858] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [14760.535154] Key type lgssc registered [14762.111350] Lustre: Echo OBD driver; http://www.lustre.org/ [14767.043818] LNet: Added LNI 192.168.202.141@tcp100 [8/256/0/180] [14768.034554] LNet: Added LNI 192.168.202.141@tcp101 [8/256/0/180] [14769.064364] LNet: Added LNI 192.168.202.141@tcp102 [8/256/0/180] [14770.194603] LNet: Added LNI 192.168.202.141@tcp103 [8/256/0/180] [14771.247561] LNet: Added LNI 192.168.202.141@tcp104 [8/256/0/180] [14772.197613] LNet: Added LNI 192.168.202.141@tcp105 [8/256/0/180] [14773.196433] LNet: Added LNI 192.168.202.141@tcp106 [8/256/0/180] [14774.142263] LNet: Added LNI 192.168.202.141@tcp107 [8/256/0/180] [14775.138981] LNet: Added LNI 192.168.202.141@tcp108 [8/256/0/180] [14776.133618] LNet: Added LNI 192.168.202.141@tcp109 [8/256/0/180] [14777.309900] LNet: Added LNI 192.168.202.141@tcp110 [8/256/0/180] [14778.359772] LNet: Added LNI 192.168.202.141@tcp111 [8/256/0/180] [14779.288982] LNet: Added LNI 192.168.202.141@tcp112 [8/256/0/180] [14780.205683] LNet: Added LNI 192.168.202.141@tcp113 [8/256/0/180] [14781.311420] LNet: Added LNI 192.168.202.141@tcp114 [8/256/0/180] [14782.368322] LNet: Added LNI 192.168.202.141@tcp115 [8/256/0/180] [14783.413876] LNet: Added LNI 192.168.202.141@tcp116 [8/256/0/180] [14784.612601] LNet: Added LNI 192.168.202.141@tcp117 [8/256/0/180] [14785.907063] LNet: Added LNI 192.168.202.141@tcp118 [8/256/0/180] [14786.994718] LNet: Added LNI 192.168.202.141@tcp119 [8/256/0/180] [14788.091227] LNet: Added LNI 192.168.202.141@tcp120 [8/256/0/180] [14789.150251] LNet: Added LNI 192.168.202.141@tcp121 [8/256/0/180] [14790.102806] LNet: Added LNI 192.168.202.141@tcp122 [8/256/0/180] [14791.195256] LNet: Added LNI 192.168.202.141@tcp123 [8/256/0/180] [14792.120603] LNet: Added LNI 192.168.202.141@tcp124 [8/256/0/180] [14793.038167] LNet: Added LNI 192.168.202.141@tcp125 [8/256/0/180] [14794.141030] LNet: Added LNI 192.168.202.141@tcp126 [8/256/0/180] [14795.421903] LNet: Added LNI 192.168.202.141@tcp127 [8/256/0/180] [14796.528545] LNet: Added LNI 192.168.202.141@tcp128 [8/256/0/180] [14797.460156] LNet: Added LNI 192.168.202.141@tcp129 [8/256/0/180] [14798.197687] LNet: Added LNI 192.168.202.141@tcp130 [8/256/0/180] [14799.038635] LNet: Added LNI 192.168.202.141@tcp131 [8/256/0/180] [14799.706312] LNet: Added LNI 192.168.202.141@tcp132 [8/256/0/180] [14800.581429] LNet: Added LNI 192.168.202.141@tcp133 [8/256/0/180] [14801.383120] LNet: Added LNI 192.168.202.141@tcp134 [8/256/0/180] [14802.027858] LNet: Added LNI 192.168.202.141@tcp135 [8/256/0/180] [14807.826200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14807.864722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14808.951602] Lustre: 463963:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.141@tcp: can't add NID 192.168.202.141@tcp130: rc = -75 [14809.212684] LNetError: 461423:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.141 [14809.244842] LNetError: Protocol error connecting to 192.168.202.141@tcp124 at host 192.168.202.141:988: is it running a compatible version of Lustre? [14809.275358] LNetError: Skipped 6 previous similar messages [14809.300338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14809.351740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14809.619161] Lustre: lustre-MDT0000: new disk, initializing [14809.750579] Lustre: 464020:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.141@tcp: can't add NID 192.168.202.141@tcp130: rc = -75 [14809.758903] Lustre: 464020:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14809.773038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14809.786734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14810.016094] LNet: 5 local NIs in recovery (showing 5): 192.168.202.141@tcp126, 192.168.202.141@tcp127, 192.168.202.141@tcp129, 192.168.202.141@tcp124, 192.168.202.141@tcp128 [14810.020873] LNetError: 463996:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.141 [14810.048227] LNetError: 463996:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14810.066812] LNetError: Protocol error connecting to 192.168.202.141@tcp106 at host 192.168.202.141:988: is it running a compatible version of Lustre? [14810.087345] LNetError: Skipped 7 previous similar messages [14816.595514] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [14834.827145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14835.138618] Lustre: 463981:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14835.322951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14835.337721] Lustre: Skipped 1 previous similar message [14835.530666] Lustre: lustre-MDT0001: new disk, initializing [14835.606925] Lustre: 464786:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.141@tcp: can't add NID 192.168.202.141@tcp130: rc = -75 [14835.623992] Lustre: 464786:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14835.956207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14836.070337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14836.098379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14839.718049] LNetError: 464006:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.141 [14839.733584] LNetError: 464006:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14839.744313] LNetError: Protocol error connecting to 192.168.202.141@tcp101 at host 192.168.202.141:988: is it running a compatible version of Lustre? [14841.892703] LNetError: 461421:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.141 [14841.926202] LNetError: Protocol error connecting to 192.168.202.141@tcp125 at host 192.168.202.141:988: is it running a compatible version of Lustre? [14842.019528] Lustre: 464992:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.141@tcp: can't add NID 192.168.202.141@tcp130: rc = -75 [14842.038680] Lustre: 464992:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14842.901343] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [14846.113489] LNetError: 464004:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.141 [14846.120554] LNetError: 464004:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14846.125765] LNetError: Protocol error connecting to 192.168.202.141@tcp126 at host 192.168.202.141:988: is it running a compatible version of Lustre? [14846.131375] LNetError: Skipped 3 previous similar messages [14853.677433] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14854.374890] LNetError: 461421:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.141 [14854.395403] LNetError: 461421:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14854.403956] LNetError: Protocol error connecting to 192.168.202.141@tcp125 at host 192.168.202.141:988: is it running a compatible version of Lustre? [14854.419603] LNetError: Skipped 6 previous similar messages [14860.116646] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14867.435841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14867.801282] Lustre: lustre-OST0000: new disk, initializing [14867.805687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14867.813502] Lustre: 465960:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14867.948638] Lustre: 465965:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.141@tcp: can't add NID 192.168.202.141@tcp130: rc = -75 [14867.957424] Lustre: 465965:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14868.017837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14870.753405] LNetError: 463984:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.141 [14870.764704] LNetError: 463984:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14870.772822] LNetError: Protocol error connecting to 192.168.202.141@tcp123 at host 192.168.202.141:988: is it running a compatible version of Lustre? [14870.796619] LNetError: Skipped 11 previous similar messages [14874.999342] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [14877.223405] Lustre: 466416:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.141@tcp: can't add NID 192.168.202.141@tcp130: rc = -75 [14877.235610] Lustre: 466416:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14877.304132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14877.315319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14877.506350] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14885.178797] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14893.027449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14893.030644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14893.032879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14893.036969] LustreError: Skipped 1 previous similar message [14893.047402] Lustre: Skipped 1 previous similar message [14898.146656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14898.154515] Lustre: Skipped 1 previous similar message [14903.264460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14903.276079] Lustre: Skipped 1 previous similar message [14905.568964] LNetError: 463985:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.141 [14905.594392] LNetError: 463985:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [14905.601826] LNetError: Protocol error connecting to 192.168.202.141@tcp125 at host 192.168.202.141:988: is it running a compatible version of Lustre? [14905.608033] LNetError: Skipped 14 previous similar messages [14907.359643] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14907.633331] Lustre: server umount lustre-OST0000 complete [14912.481855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14912.490164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14912.511049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14912.518027] Lustre: Skipped 1 previous similar message [14913.506200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14917.766587] Lustre: server umount lustre-MDT0000 complete [14921.909654] LustreError: 464785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788462946 with bad export cookie 13411619369424454477 [14921.937701] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14922.245420] Lustre: server umount lustre-MDT0001 complete [14929.674797] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [14934.500935] Key type lgssc unregistered [14935.307828] LNet: 467872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14935.320602] LNetError: 467872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14935.338253] LNet: Removed LNI 192.168.202.141@tcp [14936.373125] LNet: Removed LNI 192.168.202.141@tcp100 [14936.431588] LNetError: 467872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14936.435559] LNetError: 467872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14937.526856] LNet: Removed LNI 192.168.202.141@tcp101 [14937.554198] LNetError: 467872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14937.570714] LNet: Removed LNI 192.168.202.141@tcp102 [14938.591168] LNet: Removed LNI 192.168.202.141@tcp103 [14939.615293] LNet: Removed LNI 192.168.202.141@tcp104 [14939.638432] LNetError: 467872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14939.657370] LNetError: 467872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14939.696316] LNet: Removed LNI 192.168.202.141@tcp105 [14940.767168] LNet: Removed LNI 192.168.202.141@tcp106 [14941.791216] LNet: Removed LNI 192.168.202.141@tcp107 [14942.815295] LNet: Removed LNI 192.168.202.141@tcp108 [14943.840722] LNet: Removed LNI 192.168.202.141@tcp109 [14943.851033] LNetError: 467872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14943.861596] LNetError: 467872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14944.927139] LNet: Removed LNI 192.168.202.141@tcp110 [14945.952578] LNet: Removed LNI 192.168.202.141@tcp111 [14946.977279] LNet: Removed LNI 192.168.202.141@tcp112 [14947.999441] LNet: Removed LNI 192.168.202.141@tcp113 [14949.023196] LNet: Removed LNI 192.168.202.141@tcp114 [14950.050520] LNet: Removed LNI 192.168.202.141@tcp115 [14951.072922] LNet: Removed LNI 192.168.202.141@tcp116 [14952.095492] LNet: Removed LNI 192.168.202.141@tcp117 [14952.112536] LNetError: 467872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14952.119537] LNetError: 467872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [14952.127874] LNet: Removed LNI 192.168.202.141@tcp118 [14953.183149] LNet: Removed LNI 192.168.202.141@tcp119 [14954.207250] LNet: Removed LNI 192.168.202.141@tcp120 [14955.295165] LNet: Removed LNI 192.168.202.141@tcp121 [14956.319707] LNet: Removed LNI 192.168.202.141@tcp122 [14957.343161] LNet: Removed LNI 192.168.202.141@tcp123 [14958.431163] LNet: Removed LNI 192.168.202.141@tcp124 [14959.455747] LNet: Removed LNI 192.168.202.141@tcp125 [14960.479481] LNet: Removed LNI 192.168.202.141@tcp126 [14961.506890] LNet: Removed LNI 192.168.202.141@tcp127 [14962.527264] LNet: Removed LNI 192.168.202.141@tcp128 [14963.551138] LNet: Removed LNI 192.168.202.141@tcp129 [14963.563706] LNet: Removed LNI 192.168.202.141@tcp130 [14963.574185] LNet: Removed LNI 192.168.202.141@tcp131 [14963.589296] LNet: Removed LNI 192.168.202.141@tcp132 [14963.597718] LNet: Removed LNI 192.168.202.141@tcp133 [14963.604663] LNet: Removed LNI 192.168.202.141@tcp134 [14963.624521] LNet: Removed LNI 192.168.202.141@tcp135 [14964.651375] Key type .llcrypt unregistered [14964.654332] Key type ._llcrypt unregistered [14995.054082] Key type ._llcrypt registered [14995.057385] Key type .llcrypt registered [14995.253727] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [15012.193947] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [15013.281154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15013.321132] alg: No test for adler32 (adler32-zlib) [15014.558383] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [15014.986632] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [15016.727601] Key type lgssc registered [15019.294937] Lustre: Echo OBD driver; http://www.lustre.org/ [15062.033509] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 15:18:04 (1788463084) [15088.569898] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [15091.463512] Key type lgssc unregistered [15091.768311] LNet: 471958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15091.774082] LNetError: 471958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15091.786967] LNet: Removed LNI 192.168.202.141@tcp [15092.598596] Key type .llcrypt unregistered [15092.600537] Key type ._llcrypt unregistered [15112.222421] Key type ._llcrypt registered [15112.224820] Key type .llcrypt registered [15112.330289] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [15113.511717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15113.527878] alg: No test for adler32 (adler32-zlib) [15114.574089] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [15114.793515] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [15116.479278] Key type lgssc registered [15118.496411] Lustre: Echo OBD driver; http://www.lustre.org/ [15130.585758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15130.639611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15132.177304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15132.233406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15132.426380] Lustre: lustre-MDT0000: new disk, initializing [15132.585607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15132.603112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15139.679032] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [15152.700624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15152.890463] Lustre: 473169:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15152.987741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15152.998202] Lustre: Skipped 1 previous similar message [15153.151987] Lustre: lustre-MDT0001: new disk, initializing [15153.292685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15153.323775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15153.346136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15158.800388] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [15169.707838] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15176.341123] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15184.277298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15184.658964] Lustre: lustre-OST0000: new disk, initializing [15184.661923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15184.669660] Lustre: 475140:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15184.782236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15190.137705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15190.156082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15190.295907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15192.255851] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [15201.302365] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15205.172471] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing lnet_if_list [15210.718218] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15210.747789] LNet: Added LNI 192.168.202.141@tcp10 [8/256/0/180] [15217.771443] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15217.816787] LNet: Added LNI 192.168.202.141@tcp11 [8/256/0/180] [15223.853113] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15223.891447] LNet: Added LNI 192.168.202.141@tcp12 [8/256/0/180] [15228.914412] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15228.975906] LNet: Added LNI 192.168.202.141@tcp13 [8/256/0/180] [15234.286110] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15234.319119] LNet: Added LNI 192.168.202.141@tcp14 [8/256/0/180] [15239.449594] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15239.506456] LNet: Added LNI 192.168.202.141@tcp15 [8/256/0/180] [15245.131367] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15245.163908] LNet: Added LNI 192.168.202.141@tcp16 [8/256/0/180] [15250.651933] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15250.709596] LNet: Added LNI 192.168.202.141@tcp17 [8/256/0/180] [15257.102971] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15257.152762] LNet: Added LNI 192.168.202.141@tcp18 [8/256/0/180] [15263.571070] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15263.626342] LNet: Added LNI 192.168.202.141@tcp19 [8/256/0/180] [15269.362845] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15269.382780] LNet: Added LNI 192.168.202.141@tcp20 [8/256/0/180] [15275.265430] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15275.292689] LNet: Added LNI 192.168.202.141@tcp21 [8/256/0/180] [15281.267779] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15281.324555] LNet: Added LNI 192.168.202.141@tcp22 [8/256/0/180] [15290.444940] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15290.481710] LNet: Added LNI 192.168.202.141@tcp23 [8/256/0/180] [15297.924667] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15297.975371] LNet: Added LNI 192.168.202.141@tcp24 [8/256/0/180] [15305.386033] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15305.414627] LNet: Added LNI 192.168.202.141@tcp25 [8/256/0/180] [15313.599631] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15313.629534] LNet: Added LNI 192.168.202.141@tcp26 [8/256/0/180] [15321.108719] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15321.123755] LNet: Added LNI 192.168.202.141@tcp27 [8/256/0/180] [15327.725713] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15327.749963] LNet: Added LNI 192.168.202.141@tcp28 [8/256/0/180] [15335.055346] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15335.100067] LNet: Added LNI 192.168.202.141@tcp29 [8/256/0/180] [15344.524199] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15344.548826] LNet: Added LNI 192.168.202.141@tcp30 [8/256/0/180] [15352.167689] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15352.194361] LNet: Added LNI 192.168.202.141@tcp31 [8/256/0/180] [15360.524220] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15360.599357] LNet: Added LNI 192.168.202.141@tcp32 [8/256/0/180] [15370.590550] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15370.632915] LNet: Added LNI 192.168.202.141@tcp33 [8/256/0/180] [15379.619753] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15379.651934] LNet: Added LNI 192.168.202.141@tcp34 [8/256/0/180] [15386.530958] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15386.574591] LNet: Added LNI 192.168.202.141@tcp35 [8/256/0/180] [15395.705452] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15395.764644] LNet: Added LNI 192.168.202.141@tcp36 [8/256/0/180] [15405.630099] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15405.683244] LNet: Added LNI 192.168.202.141@tcp37 [8/256/0/180] [15414.934680] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15414.978585] LNet: Added LNI 192.168.202.141@tcp38 [8/256/0/180] [15423.572100] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15423.613039] LNet: Added LNI 192.168.202.141@tcp39 [8/256/0/180] [15431.671592] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15431.714905] LNet: Added LNI 192.168.202.141@tcp40 [8/256/0/180] [15440.078022] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15440.102981] LNet: Added LNI 192.168.202.141@tcp41 [8/256/0/180] [15449.739033] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15449.766695] LNet: Added LNI 192.168.202.141@tcp42 [8/256/0/180] [15459.070729] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15459.138337] LNet: Added LNI 192.168.202.141@tcp43 [8/256/0/180] [15470.044266] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15470.068991] LNet: Added LNI 192.168.202.141@tcp44 [8/256/0/180] [15480.622082] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15480.663248] LNet: Added LNI 192.168.202.141@tcp45 [8/256/0/180] [15490.895373] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15490.920934] LNet: Added LNI 192.168.202.141@tcp46 [8/256/0/180] [15497.822905] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15497.853128] LNet: Added LNI 192.168.202.141@tcp47 [8/256/0/180] [15509.476618] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15509.500478] LNet: Added LNI 192.168.202.141@tcp48 [8/256/0/180] [15521.547575] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15521.587763] LNet: Added LNI 192.168.202.141@tcp49 [8/256/0/180] [15533.196956] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15533.242416] LNet: Added LNI 192.168.202.141@tcp50 [8/256/0/180] [15554.018947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15554.026491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15554.039711] Lustre: Skipped 2 previous similar messages [15554.048489] Lustre: Skipped 1 previous similar message [15559.142550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15559.153170] Lustre: Skipped 2 previous similar messages [15564.260351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15568.351231] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15568.689897] Lustre: server umount lustre-MDT0000 complete [15573.441217] LustreError: 473161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463597 with bad export cookie 4514678878018333932 [15573.452163] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15573.454624] LustreError: 473161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15573.901924] Lustre: server umount lustre-MDT0001 complete [15589.855171] Lustre: 472443:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788463598/real 1788463598] req@00000000738a13bd x1875339523580672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788463614 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15589.883411] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15594.015090] Lustre: 472443:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788463602/real 1788463602] req@00000000b1f99dcd x1875339523580800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788463618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15596.056270] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [15599.073120] Lustre: 472440:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788463607/real 1788463607] req@0000000067a445c7 x1875339523581184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788463623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15599.104129] Lustre: 472440:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15605.217481] Lustre: 472443:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788463613/real 1788463613] req@00000000b62f807d x1875339523581312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788463629 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15606.241431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15618.543987] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3ea75c850f3df70a [15618.572427] Lustre: MGC192.168.202.141@tcp: Connection restored to 0@lo (at 0@lo) [15618.928893] LustreError: 475134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15618.945481] LustreError: 475134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15619.081084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15620.074733] LustreError: 475135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15620.118414] LustreError: 475135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15624.169558] LustreError: 475134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15624.170553] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15625.338381] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [15629.286168] LustreError: 475134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15634.223963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15634.404351] LustreError: 475135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15634.733856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15635.769732] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15640.327431] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [15647.315414] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15653.069608] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15680.481801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15680.502413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15680.513787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15680.524444] Lustre: Skipped 2 previous similar messages [15680.995876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15687.136660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15687.148769] Lustre: Skipped 4 previous similar messages [15692.255204] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15692.771327] Lustre: server umount lustre-OST0000 complete [15701.475114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15701.488869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15701.531496] Lustre: Skipped 1 previous similar message [15701.553931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15701.569501] Lustre: Skipped 2 previous similar messages [15704.105783] Lustre: server umount lustre-MDT0000 complete [15707.638487] LustreError: 499396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15708.434629] LustreError: 473162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463732 with bad export cookie 4514678878018336522 [15708.458481] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15709.022467] Lustre: server umount lustre-MDT0001 complete [15722.128994] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [15725.812434] Key type lgssc unregistered [15726.354751] LNet: 502197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15726.364664] LNetError: 502197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15726.370892] LNet: Removed LNI 192.168.202.141@tcp [15726.373560] LNet: Removed LNI 192.168.202.141@tcp10 [15726.376360] LNet: Removed LNI 192.168.202.141@tcp11 [15726.378783] LNet: Removed LNI 192.168.202.141@tcp12 [15726.381645] LNet: Removed LNI 192.168.202.141@tcp13 [15726.384645] LNet: Removed LNI 192.168.202.141@tcp14 [15726.387483] LNet: Removed LNI 192.168.202.141@tcp15 [15726.390314] LNet: Removed LNI 192.168.202.141@tcp16 [15726.392423] LNet: Removed LNI 192.168.202.141@tcp17 [15726.397810] LNet: Removed LNI 192.168.202.141@tcp18 [15726.400348] LNet: Removed LNI 192.168.202.141@tcp19 [15726.402680] LNet: Removed LNI 192.168.202.141@tcp20 [15726.405550] LNet: Removed LNI 192.168.202.141@tcp21 [15726.410950] LNet: Removed LNI 192.168.202.141@tcp22 [15726.418714] LNet: Removed LNI 192.168.202.141@tcp23 [15726.431893] LNet: Removed LNI 192.168.202.141@tcp24 [15726.440897] LNet: Removed LNI 192.168.202.141@tcp25 [15726.448574] LNet: Removed LNI 192.168.202.141@tcp26 [15726.461893] LNet: Removed LNI 192.168.202.141@tcp27 [15726.463401] LNet: Removed LNI 192.168.202.141@tcp28 [15726.470601] LNet: Removed LNI 192.168.202.141@tcp29 [15726.473611] LNet: Removed LNI 192.168.202.141@tcp30 [15726.479330] LNet: Removed LNI 192.168.202.141@tcp31 [15726.483477] LNet: Removed LNI 192.168.202.141@tcp32 [15726.490319] LNet: Removed LNI 192.168.202.141@tcp33 [15726.494527] LNet: Removed LNI 192.168.202.141@tcp34 [15726.500759] LNet: Removed LNI 192.168.202.141@tcp35 [15726.504306] LNet: Removed LNI 192.168.202.141@tcp36 [15726.507389] LNet: Removed LNI 192.168.202.141@tcp37 [15726.509512] LNet: Removed LNI 192.168.202.141@tcp38 [15726.511727] LNet: Removed LNI 192.168.202.141@tcp39 [15726.515535] LNet: Removed LNI 192.168.202.141@tcp40 [15726.518864] LNet: Removed LNI 192.168.202.141@tcp41 [15726.526253] LNet: Removed LNI 192.168.202.141@tcp42 [15726.528694] LNet: Removed LNI 192.168.202.141@tcp43 [15726.531373] LNet: Removed LNI 192.168.202.141@tcp44 [15726.534915] LNet: Removed LNI 192.168.202.141@tcp45 [15726.536971] LNet: Removed LNI 192.168.202.141@tcp46 [15726.539275] LNet: Removed LNI 192.168.202.141@tcp47 [15726.541664] LNet: Removed LNI 192.168.202.141@tcp48 [15726.543948] LNet: Removed LNI 192.168.202.141@tcp49 [15726.552749] LNet: Removed LNI 192.168.202.141@tcp50 [15727.578779] Key type .llcrypt unregistered [15727.580505] Key type ._llcrypt unregistered [15753.651712] Key type ._llcrypt registered [15753.655555] Key type .llcrypt registered [15753.771200] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [15774.537514] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [15776.194195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15776.386598] alg: No test for adler32 (adler32-zlib) [15777.536775] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [15777.885254] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [15779.615335] Key type lgssc registered [15781.155140] Lustre: Echo OBD driver; http://www.lustre.org/ [15823.797388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15823.850559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15825.217804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15825.273628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15825.377194] Lustre: lustre-MDT0000: new disk, initializing [15825.462855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15825.532907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15831.844847] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [15843.063881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15843.230789] Lustre: 505393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15843.263931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15843.266710] Lustre: Skipped 1 previous similar message [15843.343903] Lustre: lustre-MDT0001: new disk, initializing [15843.483322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15843.535489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15843.541535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15847.766773] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [15859.677472] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15866.815770] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15879.423069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15879.618808] Lustre: *** cfs_fail_loc=90f, val=0*** [15879.681659] Lustre: lustre-OST0000: new disk, initializing [15879.684936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15879.693232] Lustre: 507413:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15879.776180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15887.471580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15887.480484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15887.681834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15887.994371] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [15900.378305] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15913.443039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15913.445227] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15913.470160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15913.479459] Lustre: Skipped 1 previous similar message [15914.810080] Lustre: server umount lustre-OST0000 complete [15923.680521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15923.695374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15923.710452] Lustre: Skipped 2 previous similar messages [15923.736025] Lustre: Skipped 1 previous similar message [15927.159060] Lustre: server umount lustre-MDT0000 complete [15932.245710] LustreError: 506189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463956 with bad export cookie 6569050477928572533 [15932.248573] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15932.254380] LustreError: 506189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15933.203775] Lustre: server umount lustre-MDT0001 complete [15947.569365] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [15951.265683] Key type lgssc unregistered [15951.566342] LNet: 509318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15951.576560] LNetError: 509318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15952.621472] LNet: Removed LNI 192.168.202.141@tcp [15954.091992] Key type .llcrypt unregistered [15954.095551] Key type ._llcrypt unregistered [15981.088304] Key type ._llcrypt registered [15981.093877] Key type .llcrypt registered [15981.260623] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [15983.275470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15983.383707] alg: No test for adler32 (adler32-zlib) [15984.464621] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [15984.857410] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [15986.535172] Key type lgssc registered [15988.563833] Lustre: Echo OBD driver; http://www.lustre.org/ [16000.665791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16000.693344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16002.443900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16008.856051] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [16012.779376] LustreError: 510589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16012.821708] LustreError: 510589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16017.905437] LustreError: 510588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16021.113592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16021.982723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16028.584832] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [16038.546458] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16045.792974] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16054.021647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16054.512458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16061.931806] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [16070.090216] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16086.832277] loop0: detected capacity change from 0 to 204800000 [16087.053222] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16087.080290] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16087.112874] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16094.629546] loop0: detected capacity change from 0 to 204800000 [16094.850261] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16094.881723] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16094.903890] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16101.548567] loop0: detected capacity change from 0 to 204800000 [16107.537837] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16107.834823] Lustre: fs15246-OST0000: new disk, initializing [16107.838728] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16107.848550] Lustre: 514185:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16107.931401] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16115.473455] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [16124.462416] loop1: detected capacity change from 0 to 204800000 [16130.666516] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16130.868446] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16130.972455] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16131.113726] Lustre: fs15246-MDT0000: new disk, initializing [16131.261763] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16131.285482] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16131.404672] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16132.460617] Lustre: 515306:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b6159952 x1875340436751616/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788464172 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16135.908718] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [16141.606540] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16141.630402] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16141.672916] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16161.276567] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16161.320149] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16163.566803] Lustre: server umount fs15246-MDT0000 complete [16166.372256] LustreError: 513190:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16171.503344] LustreError: 510584:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16173.272034] Lustre: server umount fs15246-OST0000 complete [16184.800654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16184.811432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16184.830891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16184.833574] Lustre: Skipped 1 previous similar message [16186.849336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16186.867676] Lustre: Skipped 1 previous similar message [16191.968986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16197.096105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16197.111212] Lustre: Skipped 2 previous similar messages [16199.135680] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16199.419996] Lustre: server umount lustre-OST0000 complete [16206.304667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16206.314609] LustreError: Skipped 1 previous similar message [16206.322168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16206.342918] Lustre: Skipped 1 previous similar message [16206.366707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16210.112760] Lustre: server umount lustre-MDT0000 complete [16212.454333] LustreError: 510589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16212.474673] LustreError: 510589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16214.046728] LustreError: 510569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464238 with bad export cookie 6022272830368627024 [16214.053142] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16214.063486] LustreError: 510569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16214.656558] Lustre: server umount lustre-MDT0001 complete [16228.225669] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [16231.693803] Key type lgssc unregistered [16232.056807] LNet: 518403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16232.060945] LNetError: 518403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16232.072540] LNet: Removed LNI 192.168.202.141@tcp [16233.124473] Key type .llcrypt unregistered [16233.130149] Key type ._llcrypt unregistered [16256.781377] Key type ._llcrypt registered [16256.785172] Key type .llcrypt registered [16256.941002] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [16258.450685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16258.478302] alg: No test for adler32 (adler32-zlib) [16259.640259] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [16260.021815] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [16261.791216] Key type lgssc registered [16263.169889] Lustre: Echo OBD driver; http://www.lustre.org/ [16304.667327] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [16316.918592] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [16353.520470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16353.575372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16354.797892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16354.816851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16354.866720] Lustre: lustre-MDT0000: new disk, initializing [16354.920291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16354.935587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16358.952786] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [16370.016632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16370.143746] Lustre: 522888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16370.223982] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16370.225815] Lustre: Skipped 1 previous similar message [16370.338484] Lustre: lustre-MDT0001: new disk, initializing [16370.441742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16370.507427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16370.526396] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16376.067725] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [16386.793969] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16394.594061] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16403.951319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16404.397981] Lustre: lustre-OST0000: new disk, initializing [16404.405321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16404.425296] Lustre: 524857:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16404.601167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16412.296813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16412.320216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16412.449239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16412.869426] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [16422.742714] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16427.273735] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16427.463488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16431.978619] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16432.236494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16438.242274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16438.246961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16438.261485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16438.266970] Lustre: Skipped 1 previous similar message [16438.270723] LustreError: Skipped 1 previous similar message [16440.350255] Lustre: server umount lustre-OST0000 complete [16443.363277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16443.374315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16443.377303] Lustre: Skipped 1 previous similar message [16443.380663] Lustre: Skipped 1 previous similar message [16448.486583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16448.490724] Lustre: Skipped 2 previous similar messages [16449.722864] Lustre: server umount lustre-MDT0000 complete [16454.344252] LustreError: 524856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464478 with bad export cookie 808828226132743160 [16454.352204] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16454.359896] LustreError: 524856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16454.912170] Lustre: server umount lustre-MDT0001 complete [16487.605654] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [16498.776664] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [16534.088980] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16248 sec ========= 15:42:36 (1788464556) [16536.433183] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:42:38 (1788464558) === [16540.753990] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:42:43 (1788464563) === [16585.695836] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [16588.575924] Key type lgssc unregistered [16588.954907] LNet: 531505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16588.961373] LNetError: 531505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16589.997159] LNet: Removed LNI 192.168.202.141@tcp [16591.247948] Key type .llcrypt unregistered [16591.250381] Key type ._llcrypt unregistered